gcp4al19gr417a$ ps -ef | grep introscopegr417a 1944 32527 0 0626 pts9 000000 grep introscopebeaadmin 4314 19818 12 0434 001414 optjdk160_22jrebinjava -DweblogicName=edbms1905 -Djavasecuritypolicy=null -Dweblogicmanagementserver=httpgcp4al27a00cifattcom8004 -Djavalibrarypath=optjdk160_22jrelibi386serveroptjdk160_22jrelibi386optjdk160_22jrelibi386optbea103wlserverservernativelinuxi686optbea103wlserverservernativelinuxi686oci920_8usrjavapackageslibi386libusrlib -Djavaclasspath=optedblibojdbc14jaroptbea103patch_wlw1030profilesdefaultsys_manifest_classpathweblogic_patchjaroptbea103patch_wls1030profilesdefaultsys_manifest_classpathweblogic_patchjaroptbea103patch_cie660profilesdefaultsys_manifest_classpathweblogic_patchjaroptjdk6libtoolsjaroptbea103wlserverserverlibweblogic_spjaroptbea103wlserverserverlibweblogicjaroptbea103modulesfeaturesweblogicservermodules_10300jaroptbea103wlserverserverlibwebservicesjaroptbea103modulesorgapacheant_165libant-alljaroptbea103modulesnetsfantcontrib_1000_1-0b2libant-contribjaroptbea103wlserverserverlibxqrljaroptedbmserveredbms1905confoptedbappwilyAgentwilyWebAppSupportjar -DweblogicsystemBootIdentityFile=optedbmserveredbms1905serversedbms1905datanodemanagerbootproperties -DweblogicnodemanagerServiceEnabled=true -DweblogicsecuritySSLignoreHostnameVerification=true -DweblogicReverseDNSAllowed=false -Djavacompiler=NONE -Xms2048m -Xmx2048m -XXCompileThreshold=8000 -XXPermSize=256m -XXMaxPermSize=256m -verbosegc -Xloggclogsgclog -XX+PrintGCDetails -XX+PrintGCTimeStamps -Dplatformhome=optbea103wlserver -Dwlshome=optbea103wlserverserver -DwlwiterativeDev=false -DwlwtestConsole=false -DweblogicName=edbms1905 -DweblogicStdout=optedbmserveredbms1905logsweblogicserverlog -DweblogicStderr=optedbmserveredbms1905logsweblogicserverlog -Dconfdir=optedbmserveredbms1905 -Dapp_root_dir=optedbapp -DSERVICELOGGER_CONF_FILE_PATH=optedbmserveredbms1905confserviceloggerxmlxml -DweblogicdeploymentHeartbeatPeriodSeconds=600 -DweblogicconsolesystemstatusdefaultMinimized=true -DweblogicclassloadernoJarSigners=true -DweblogichttpclientdefaultReadTimeout=30000 -DUseSunHttpHandler=true -javaagentoptedbappwilyAgentwilyAgentjar -DcomwilyintroscopeagentProfile=optedbappwilyAgentwilyIntroscopeAgentprofile weblogicServerbeaadmin 8236 19818 5 0442 000527 optjdk160_22jrebinjava -DweblogicName=edbms1904 -Djavasecuritypolicy=null -Dweblogicmanagementserver=httpgcp4al27a00cifattcom8004 -Djavalibrarypath=optjdk160_22jrelibi386serveroptjdk160_22jrelibi386optjdk160_22jrelibi386optbea103wlserverservernativelinuxi686optbea103wlserverservernativelinuxi686oci920_8usrjavapackageslibi386libusrlib -Djavaclasspath=optbea103patch_wlw1030profilesdefaultsys_manifest_classpathweblogic_patchjaroptbea103patch_wls1030profilesdefaultsys_manifest_classpathweblogic_patchjaroptbea103patch_cie660profilesdefaultsys_manifest_classpathweblogic_patchjaroptjdk6libtoolsjaroptbea103wlserverserverlibweblogic_spjaroptbea103wlserverserverlibweblogicjaroptbea103modulesfeaturesweblogicservermodules_10300jaroptbea103wlserverserverlibwebservicesjaroptbea103modulesorgapacheant_165libant-alljaroptbea103modulesnetsfantcontrib_1000_1-0b2libant-contribjaroptbea103wlserverserverlibxqrljaroptedbmserveredbms1904confoptedbappwilyAgentwilyWebAppSupportjar -DweblogicsystemBootIdentityFile=optedbmserveredbms1904serversedbms1904datanodemanagerbootproperties -DweblogicnodemanagerServiceEnabled=true -DweblogicsecuritySSLignoreHostnameVerification=true -DweblogicReverseDNSAllowed=false -Djavacompiler=NONE -Xms1024m -Xmx1024m -XXCompileThreshold=8000 -XXPermSize=256m -XXMaxPermSize=256m -verbosegc -Xloggclogsgclog -XX+PrintGCDetails -XX+PrintGCTimeStamps -Dplatformhome=optbea103wlserver -Dwlshome=optbea103wlserverserver -DwlwiterativeDev=false -DwlwtestConsole=false -DweblogicName=edbms1904 -DweblogicStdout=optedbmserveredbms1904logsweblogicserverlog -DweblogicStderr=optedbmserveredbms1904logsweblogicserverlog -Dconfdir=optedbmserveredbms1904 -Dapp_root_dir=optedbapp -DSERVICELOGGER_CONF_FILE_PATH=optedbmserveredbms1904confserviceloggerxmlxml -DweblogicdeploymentHeartbeatPeriodSeconds=600 -DweblogicconsolesystemstatusdefaultMinimized=true -DweblogicclassloadernoJarSigners=true -DweblogichttpclientdefaultReadTimeout=30000 -javaagentoptedbappwilyAgentwilyAgentjar -DcomwilyintroscopeagentProfile=optedbappwilyAgentwilyIntroscopeAgentprofile weblogicServergcp4al19gr417a$

use the Introscope Workstation to view and manipulate data that is stored by the Enterprise ManagerAs a managed application runs probes relay collected data to the agent The agent then collects and summarizes the data and sends it to the Enterprise Manager3The time displayed on data viewers now includes the time zone (for example PST for Pacific Standard Time) on the machine hosting the Enterprise Manager

Workstation locates where the computer locates for user Epagent Agent monitors the server

Introscope agent installs on our server and then for weblogicwebsphere we use introscope in start up class

We have to give t the introscope enterprise manager as host at starting and port to get the workstation

click Start gt Introscope gt Introscope Workstation or run Introscope WorkstationexeFor Java Web Start deployments of the Workstation open a browser to httpEMhost8081workstation

In a URL the same combination would be referred to like this httpltlocalhostgt8081workstationpage=investigatorampagent=SuperDomain|localhost|WebLogic|WebLogic20AgentIn the login dialog enter the Host name or IP addressraquo Note You can use the IP address instead of the host name only if both your client machine and the host machine support the same IP protocolthe Port number your User Name your Passwordraquo Tip The Workstation will remember the last five login attempts so if you have entered host and user information previously the Host Port and User ID fields will be drop-downs from which you can select the credentials you want to use

About the Workstation ConsoleThe Console is the default view when you start the Workstation and contains dashboards that show performance data in graphical views Dashboards are basic tools for viewing management data in IntroscopeThe Introscope Sample Management Module provides a set of sample dashboards Authorized users can create custom dashboards using the Dashboard Editormdashsee the Introscope Configuration and Administration Guide for more informationYou can have more than one Console window open at the same time To open a new Console window select Workstation gt New Console

Workstation locates where the computer locates for user Epagent Agent monitors the server

Introscope agent installs on our server and then for weblogicwebsphere we use introscope in start up class

We have to give t the introscope enterprise manager as host at starting and port to get the workstation

click Start gt Introscope gt Introscope Workstation or run Introscope WorkstationexeFor Java Web Start deployments of the Workstation open a browser to httpEMhost8081workstation

In a URL the same combination would be referred to like this httpltlocalhostgt8081workstationpage=investigatorampagent=SuperDomain|localhost|WebLogic|WebLogic20AgentIn the login dialog enter the Host name or IP addressraquo Note You can use the IP address instead of the host name only if both your client machine and the host machine support the same IP protocolthe Port number your User Name your Passwordraquo Tip The Workstation will remember the last five login attempts so if you have entered host and user information previously the Host Port and User ID fields will be drop-downs from which you can select the credentials you want to use

About the Workstation ConsoleThe Console is the default view when you start the Workstation and contains dashboards that show performance data in graphical views Dashboards are basic tools for viewing management data in IntroscopeThe Introscope Sample Management Module provides a set of sample dashboards Authorized users can create custom dashboards using the Dashboard Editormdashsee the Introscope Configuration and Administration Guide for more informationYou can have more than one Console window open at the same time To open a new Console window select Workstation gt New Console

About the Workstation InvestigatorYou use the Investigator to view application and system status to search and to browse metric data using a tree structureSelect Workstation gt New Investigator

This is java

This is net appThe Investigator contains these sections The Investigator tree provides information about each host and application managed by the Enterprise Manager The metrics that appear in the Investigator tree are a function of the resources your applications use and the data that your Introscope agents are configured to reportThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric in the tree You can select View tabs to open different views of data The tabs that are

available vary depending on the item selected in the tree For some views options might be available in the bottom section of the Viewer pane to control the data displayed in the Viewer

About the Management Module EditorYou use the Management Module Editor to create or edit a Management Module which contains a set of Introscope monitoring configuration information Management Modules are listed for each domain and contain objects known as elements that contain and organize data with monitoring logicmdashalerts actions and dashboards

The Dashboard Editor provides tools for creating and laying out Data Viewers shapes lines text boxes and connectors Users with appropriate permissions can create and edit dashboards and dashboard objects such as imported images shapes lines and textmdashsee the Introscope Configuration and Administration Guide for more information

In management module we can keep the barchart GraphicEqualizerString viewer etc hellip we can change this by right clicking in management module

In Introscope Workstation users are assigned user permissions Each Workstation user is assigned a user name password and certain permissions Permissions are granted at the Domain and Enterprise levelUser preferencesYou use Introscope user preferences to specifya home dashboard 1048708 whether to display Management Module names alongside dashboard names in

the Console low-threshold execution-time warnings for Transaction TracerTo change your home dashboard1 Select Workstation gt User Preferences

Click Choose enter a search string to narrow the selection and select from the remaining list

You can use the same name for dashboards that are in different Management Modules and use the same name for Management Modules that are in different Domains1 Select Workstation gt User Preferences2 Check Show Module and Domain name with Dashboard name3 Click ApplyThe Management Module and domain that contain the dashboard appear after the dashboard nameTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkboxNavigating among dashboards in the ConsoleDashboard drop-down listYou can select dashboards from the drop-down list at the top of the Console page You can type all or part of the dashboard name to narrow the selections in the list

Automatic hyperlinksmdashIntroscope automatically links a Data Viewer to the metric grouping it is based uponmdashthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor Similarly dashboards that contain Data Viewers based on the same metric grouping are automatically linked and you can navigate between them using the Links menuWe can done its in management module- dashboard-right click and link toCustom hyperlinksmdashYou can define custom links for dashboard items to link to other dashboards or to web pages You can define custom links if you have dashboard editing permission

See a list of available links 1048708 Select a dashboard object and selectProperties gt Links1048708 Right-click the dashboard object and select Links from the context menu

1 Click the graph in the Console to select it2 Show the minimum and maximum values in one of two ways1048708 Right-click the Data Viewer and select Show Minimum and Maximum1048708 Select Properties menu and select Show Minimum and MaximumTo open a tool tip1048708 Mouse over any element in the Workstation metrics tree or in a Data Viewer such as a point on a graphThe illustration below displays information about a particular data point in the graph showing1048708 Metric name1048708 Exact value of the metric1048708 Minmax values for the metric across the period represented by the data point (See How time range affects data points below)1048708 The count of 15-second intervals represented by the data point (See How time range affects data points below)1048708 The date and time for the data point in the graphMoving metrics to frontback in graph

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

This is java

This is net appThe Investigator contains these sections The Investigator tree provides information about each host and application managed by the Enterprise Manager The metrics that appear in the Investigator tree are a function of the resources your applications use and the data that your Introscope agents are configured to reportThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric in the tree You can select View tabs to open different views of data The tabs that are

available vary depending on the item selected in the tree For some views options might be available in the bottom section of the Viewer pane to control the data displayed in the Viewer

About the Management Module EditorYou use the Management Module Editor to create or edit a Management Module which contains a set of Introscope monitoring configuration information Management Modules are listed for each domain and contain objects known as elements that contain and organize data with monitoring logicmdashalerts actions and dashboards

The Dashboard Editor provides tools for creating and laying out Data Viewers shapes lines text boxes and connectors Users with appropriate permissions can create and edit dashboards and dashboard objects such as imported images shapes lines and textmdashsee the Introscope Configuration and Administration Guide for more information

In management module we can keep the barchart GraphicEqualizerString viewer etc hellip we can change this by right clicking in management module

In Introscope Workstation users are assigned user permissions Each Workstation user is assigned a user name password and certain permissions Permissions are granted at the Domain and Enterprise levelUser preferencesYou use Introscope user preferences to specifya home dashboard 1048708 whether to display Management Module names alongside dashboard names in

the Console low-threshold execution-time warnings for Transaction TracerTo change your home dashboard1 Select Workstation gt User Preferences

Click Choose enter a search string to narrow the selection and select from the remaining list

You can use the same name for dashboards that are in different Management Modules and use the same name for Management Modules that are in different Domains1 Select Workstation gt User Preferences2 Check Show Module and Domain name with Dashboard name3 Click ApplyThe Management Module and domain that contain the dashboard appear after the dashboard nameTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkboxNavigating among dashboards in the ConsoleDashboard drop-down listYou can select dashboards from the drop-down list at the top of the Console page You can type all or part of the dashboard name to narrow the selections in the list

Automatic hyperlinksmdashIntroscope automatically links a Data Viewer to the metric grouping it is based uponmdashthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor Similarly dashboards that contain Data Viewers based on the same metric grouping are automatically linked and you can navigate between them using the Links menuWe can done its in management module- dashboard-right click and link toCustom hyperlinksmdashYou can define custom links for dashboard items to link to other dashboards or to web pages You can define custom links if you have dashboard editing permission

See a list of available links 1048708 Select a dashboard object and selectProperties gt Links1048708 Right-click the dashboard object and select Links from the context menu

1 Click the graph in the Console to select it2 Show the minimum and maximum values in one of two ways1048708 Right-click the Data Viewer and select Show Minimum and Maximum1048708 Select Properties menu and select Show Minimum and MaximumTo open a tool tip1048708 Mouse over any element in the Workstation metrics tree or in a Data Viewer such as a point on a graphThe illustration below displays information about a particular data point in the graph showing1048708 Metric name1048708 Exact value of the metric1048708 Minmax values for the metric across the period represented by the data point (See How time range affects data points below)1048708 The count of 15-second intervals represented by the data point (See How time range affects data points below)1048708 The date and time for the data point in the graphMoving metrics to frontback in graph

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

available vary depending on the item selected in the tree For some views options might be available in the bottom section of the Viewer pane to control the data displayed in the Viewer

About the Management Module EditorYou use the Management Module Editor to create or edit a Management Module which contains a set of Introscope monitoring configuration information Management Modules are listed for each domain and contain objects known as elements that contain and organize data with monitoring logicmdashalerts actions and dashboards

The Dashboard Editor provides tools for creating and laying out Data Viewers shapes lines text boxes and connectors Users with appropriate permissions can create and edit dashboards and dashboard objects such as imported images shapes lines and textmdashsee the Introscope Configuration and Administration Guide for more information

In management module we can keep the barchart GraphicEqualizerString viewer etc hellip we can change this by right clicking in management module

In Introscope Workstation users are assigned user permissions Each Workstation user is assigned a user name password and certain permissions Permissions are granted at the Domain and Enterprise levelUser preferencesYou use Introscope user preferences to specifya home dashboard 1048708 whether to display Management Module names alongside dashboard names in

the Console low-threshold execution-time warnings for Transaction TracerTo change your home dashboard1 Select Workstation gt User Preferences

Click Choose enter a search string to narrow the selection and select from the remaining list

You can use the same name for dashboards that are in different Management Modules and use the same name for Management Modules that are in different Domains1 Select Workstation gt User Preferences2 Check Show Module and Domain name with Dashboard name3 Click ApplyThe Management Module and domain that contain the dashboard appear after the dashboard nameTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkboxNavigating among dashboards in the ConsoleDashboard drop-down listYou can select dashboards from the drop-down list at the top of the Console page You can type all or part of the dashboard name to narrow the selections in the list

Automatic hyperlinksmdashIntroscope automatically links a Data Viewer to the metric grouping it is based uponmdashthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor Similarly dashboards that contain Data Viewers based on the same metric grouping are automatically linked and you can navigate between them using the Links menuWe can done its in management module- dashboard-right click and link toCustom hyperlinksmdashYou can define custom links for dashboard items to link to other dashboards or to web pages You can define custom links if you have dashboard editing permission

See a list of available links 1048708 Select a dashboard object and selectProperties gt Links1048708 Right-click the dashboard object and select Links from the context menu

1 Click the graph in the Console to select it2 Show the minimum and maximum values in one of two ways1048708 Right-click the Data Viewer and select Show Minimum and Maximum1048708 Select Properties menu and select Show Minimum and MaximumTo open a tool tip1048708 Mouse over any element in the Workstation metrics tree or in a Data Viewer such as a point on a graphThe illustration below displays information about a particular data point in the graph showing1048708 Metric name1048708 Exact value of the metric1048708 Minmax values for the metric across the period represented by the data point (See How time range affects data points below)1048708 The count of 15-second intervals represented by the data point (See How time range affects data points below)1048708 The date and time for the data point in the graphMoving metrics to frontback in graph

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

In Introscope Workstation users are assigned user permissions Each Workstation user is assigned a user name password and certain permissions Permissions are granted at the Domain and Enterprise levelUser preferencesYou use Introscope user preferences to specifya home dashboard 1048708 whether to display Management Module names alongside dashboard names in

the Console low-threshold execution-time warnings for Transaction TracerTo change your home dashboard1 Select Workstation gt User Preferences

Click Choose enter a search string to narrow the selection and select from the remaining list

You can use the same name for dashboards that are in different Management Modules and use the same name for Management Modules that are in different Domains1 Select Workstation gt User Preferences2 Check Show Module and Domain name with Dashboard name3 Click ApplyThe Management Module and domain that contain the dashboard appear after the dashboard nameTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkboxNavigating among dashboards in the ConsoleDashboard drop-down listYou can select dashboards from the drop-down list at the top of the Console page You can type all or part of the dashboard name to narrow the selections in the list

Automatic hyperlinksmdashIntroscope automatically links a Data Viewer to the metric grouping it is based uponmdashthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor Similarly dashboards that contain Data Viewers based on the same metric grouping are automatically linked and you can navigate between them using the Links menuWe can done its in management module- dashboard-right click and link toCustom hyperlinksmdashYou can define custom links for dashboard items to link to other dashboards or to web pages You can define custom links if you have dashboard editing permission

See a list of available links 1048708 Select a dashboard object and selectProperties gt Links1048708 Right-click the dashboard object and select Links from the context menu

1 Click the graph in the Console to select it2 Show the minimum and maximum values in one of two ways1048708 Right-click the Data Viewer and select Show Minimum and Maximum1048708 Select Properties menu and select Show Minimum and MaximumTo open a tool tip1048708 Mouse over any element in the Workstation metrics tree or in a Data Viewer such as a point on a graphThe illustration below displays information about a particular data point in the graph showing1048708 Metric name1048708 Exact value of the metric1048708 Minmax values for the metric across the period represented by the data point (See How time range affects data points below)1048708 The count of 15-second intervals represented by the data point (See How time range affects data points below)1048708 The date and time for the data point in the graphMoving metrics to frontback in graph

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

the Console low-threshold execution-time warnings for Transaction TracerTo change your home dashboard1 Select Workstation gt User Preferences

Click Choose enter a search string to narrow the selection and select from the remaining list

You can use the same name for dashboards that are in different Management Modules and use the same name for Management Modules that are in different Domains1 Select Workstation gt User Preferences2 Check Show Module and Domain name with Dashboard name3 Click ApplyThe Management Module and domain that contain the dashboard appear after the dashboard nameTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkboxNavigating among dashboards in the ConsoleDashboard drop-down listYou can select dashboards from the drop-down list at the top of the Console page You can type all or part of the dashboard name to narrow the selections in the list

Automatic hyperlinksmdashIntroscope automatically links a Data Viewer to the metric grouping it is based uponmdashthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor Similarly dashboards that contain Data Viewers based on the same metric grouping are automatically linked and you can navigate between them using the Links menuWe can done its in management module- dashboard-right click and link toCustom hyperlinksmdashYou can define custom links for dashboard items to link to other dashboards or to web pages You can define custom links if you have dashboard editing permission

See a list of available links 1048708 Select a dashboard object and selectProperties gt Links1048708 Right-click the dashboard object and select Links from the context menu

1 Click the graph in the Console to select it2 Show the minimum and maximum values in one of two ways1048708 Right-click the Data Viewer and select Show Minimum and Maximum1048708 Select Properties menu and select Show Minimum and MaximumTo open a tool tip1048708 Mouse over any element in the Workstation metrics tree or in a Data Viewer such as a point on a graphThe illustration below displays information about a particular data point in the graph showing1048708 Metric name1048708 Exact value of the metric1048708 Minmax values for the metric across the period represented by the data point (See How time range affects data points below)1048708 The count of 15-second intervals represented by the data point (See How time range affects data points below)1048708 The date and time for the data point in the graphMoving metrics to frontback in graph

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Automatic hyperlinksmdashIntroscope automatically links a Data Viewer to the metric grouping it is based uponmdashthe Links menu for the viewer contains a link to the underlying metric grouping definition in the Management Module Editor Similarly dashboards that contain Data Viewers based on the same metric grouping are automatically linked and you can navigate between them using the Links menuWe can done its in management module- dashboard-right click and link toCustom hyperlinksmdashYou can define custom links for dashboard items to link to other dashboards or to web pages You can define custom links if you have dashboard editing permission

See a list of available links 1048708 Select a dashboard object and selectProperties gt Links1048708 Right-click the dashboard object and select Links from the context menu

1 Click the graph in the Console to select it2 Show the minimum and maximum values in one of two ways1048708 Right-click the Data Viewer and select Show Minimum and Maximum1048708 Select Properties menu and select Show Minimum and MaximumTo open a tool tip1048708 Mouse over any element in the Workstation metrics tree or in a Data Viewer such as a point on a graphThe illustration below displays information about a particular data point in the graph showing1048708 Metric name1048708 Exact value of the metric1048708 Minmax values for the metric across the period represented by the data point (See How time range affects data points below)1048708 The count of 15-second intervals represented by the data point (See How time range affects data points below)1048708 The date and time for the data point in the graphMoving metrics to frontback in graph

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

When a graph contains multiple metrics it is possible for data points to overlay each other You can use the Bring to Front or Send to Back options to choose which metric appears at the top of the list of metrics

To view the scale of a graph chart 1048708 Click on a chart to select it and then 1048708 Select Viewer gt Scale Options or 1048708 Right-click the chart and select Scale Options from the context menu The Data Options dialog box opens

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

To force minimum and maximum values1 Click on a chart to select it 2 Select Viewer gt Scale Options3 Select Pin at on both the Minimum and Maximum sides of the dialog and enter a value for the minimum and maximum points of the data access 4 Click OKFor example if the chart data values lie primarily between 350 and 550 but the chart value axis shows 0-1000 it might be helpful to set the scale Min value to 300 and Max value to 600 for a better view of the relevant data

1 Click on a chart to select it2 Select Viewer gt Scale Options3 Choose Auto Expand on both the Minimum and Maximum side of the dialog 4 Click OK

2 Select Viewer gt Scale Options3 Select AutoScale on both the Minimum and Maximum sides of the dialogThe resulting chartrsquos data axis is reset based on the data in the chart as shown in the illustration below This often results in sharper valleys and peaks in the graph display

To copy a Data Viewer to the clipboard1 Open a Console and select a Data Viewer2 Select Viewer gt Copy to Clipboard as ImageTo export data from a Data Viewer1 In the Console select a Data Viewer2 Select Viewer gt Export DataYou can view live data in the Console or select a range of time to view historical data The default view of data is Live

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

To view historical data1 Select the metric or dashboard for which you want to see historical data2 Select a time range for the historical view from the Time Range drop-down menu

Now you can select a resolution to adjust the granularity of the view by increasing or decreasing the number of data points that appearEach pre-defined time range is associated with a default resolution You normally do not need to change this Changing the resolution is useful when you need to see a greater level of detail or granularity in the data than appears by default For 8 mins its 15 seconds 1 day its 12 mins seems the data its showing spike in that time only

Drag the slider on the time bar to change the time range

Click the arrows to move backward and forward in timeThe single arrows move backward or forward in small increments the doublearrows move backward or forward in time increments that are about equal tothe time of the selected time range Click the Reset icon to reset the end time of the range to the current time

Select Custom Range from the Time Range drop-down menu

Use the calendar controls to select the start and end dates and times

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

When you view historical data in a graph you can zoom in on data by clicking the mouse pointer on a graph position and dragging to specify the time range

Set the global time range and the Time Range control to match the zoomed view select Viewer gt Set Time Range From Zoomed Range or click the Set Time Range from Zoomed Range icon Lock your selected resolution by clicking the Lock icon This maintains your selected resolution as you select different time ranges by zooming in on dataHold down the shift key while you zoom to constrain zooming to the time axis

You use the Console Lens to filter metric data for the agents that are reporting data In a dashboard that shows data for more than one agent you can use the Console Lens to view data only for selected agents

Click the Lens button or select Dashboard gt Lens

If you select and click apply it would displays only that data in the console and donrsquot show for other agents or process Again make it to all by clicking remove

In the Select Agent dialog box select a single agent or select multiple agents (click and drag or CTRLclick) on which to filterClick Apply or press Enter The dashboard refreshes to show only data for the selected agent(s) The Lens button shows a black arrowTo clear the Console Lens1 Click Lens2 Clear the Lens by clicking the Clear button on the Apply Agent Lens dialog box

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Workstation InvestigatorThe Investigator contains two main panes Investigator TreemdashThe left side of the Investigator is a tree structure that organizes metric data reported by the agents that report to the Enterprise Manager you are logged into Metrics are organized hierarchically by host process agent and resource type Selecting items in the tree causespredefined views to be presented in the Viewer pane Inactive metrics appear grayed out in the treeViewermdashThe Viewer pane on the right side of the Investigator presents details often graphical for the resource or metric currently selected in the tree Depending on the item selected in the tree tabs at the top of the Viewer pane allow you to select one or more views including the General Overview Errors Traces and Search views For some views options might be available in the bottom section of the Viewer pane that control the data displayed in the Viewer

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

The SuperDomain node contains metrics for all agents that report to the Enterprise Manager to which the Workstation is connected Metrics are organized in a Host|Process|Agent hierarchy The nodes immediately under the SuperDomain node are virtual and physicalhostsCustom Metric Host (Virtual)mdashThis node does not correspond to a physical host machine It is a virtual host that contains metrics that are not reported by a specific individual agent For example if you have configured calculators that create custom metrics or have configured aggregated agents they typically appear under the Custom Metric Host HostsmdashOne node for each machine that hosts an agent Each host node contains a process node for the instance of the application being monitored which in turn contains an agent node The agent node contains nodes that correspond to application and system resources which contain metricsThe SuperDomain is that which includes all user-defined domains and agents The Enterprise Manager administrator can set up the EM to display child domains with separate permissionsThis illustration shows two child domains myDomain1 and myDomain2 listed under the Domains node as well as under the SuperDomain node in the default Custom Metric Process

Supportability metricsSupportability metrics give information about the state of the Enterprise Manager and the machine it runs on You can view them under the path SuperDomain|Custom Metric Host|Custom Metric Agent|Enterprise Manager The Introscope Sizing Guide contains extensive information about the supportability metrics

To change Data Viewer type1 Select Properties gt View As2 Select the viewer display type to change toTop ten viewsWhen you select certain resources in the Investigator the General tab of the Viewer pane shows the top ten matching metrics for the selected resource Java resources include servlets JSP EJBs and JDBC for NET resources include ASPNET ADONET and serviced componentsThese metrics appear in a bar chart in the Investigator Preview pane

Select Workstation gt New Investigator

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

General tab When you select a metric the General tab shows a graphic view of the metricmdasheither for live data or for a selected historical period See Viewing historical data on page 87 for an explanation of how to select ranges of historical data to view For some nodes in the tree the General tab shows the path to that node object in the Investigator hierarchy For example when the Frontends node is selected the General tab shows this path

Application OverviewThe Application Overview is available when you select an agent in the Investigator tree and enables application monitoring and triage It shows highlevel health indicators and a log of related events and historical metric informationThe Overview shows a row of lights for each application managed by the currently selected agent Introscope presents this data for each application it discoversmdash when a servlet executes Introscope makes a call to getServletContextName() of the ServletContext interface to determine the name of the application After the application starts the Overview tab automatically updates to display a row oflights for it

This illustration shows one application -- its name is truncated to CatalogServletCont -- managed by this agent For this application you can view alerts showing the state ofDouble-clicking one of the alert indicators under the User column links to the URL metrics for the selected applicationUser Indicates how satisfactory the end-usersrsquo interactions with the application are likely to be Satisfaction is a function of response time waits stalls and errors1048708 Greenmdashnormal satisfactory user interactions with the application1048708 Yellowmdashan attempt to use the application is likely to yieldunsatisfactory results for instance poor response time or errors1048708 Redmdashindicates a serious availability issue and that an attempt to use the application will probably fail

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

VM Indicates the health and availability of server resources such as resource pools and CPU1048708 Greenmdashnormal health of server resources1048708 Yellowmdashresource limitations or outages1048708 Redmdashserious resource limitations or outagesBackend SummaryIndicates the worst health and availability across all backends accessed by the application For example if one of three backends has a serious resource limitation or outage the All Backends light is red The purposeof the All Backends light is to allow th e user with minimal scrolling to quickly assess whether any of the backends have problems that require investigation1048708 Greenmdashnormal backend health and availability across all backends accessed by the application1048708 Yellowmdashat least one backend accessed by the application is experiencing errors or stalls or poorer than expected response times1048708 Redmdashat least one backend accessed by theUsing alerts to drill down for more dataYou can double-click an alert from the overview tab to display the underlying data for that application tier For example if you double-click the User alert the Workstation will display the URLs node for that agent

If you click VM in the topmost screen it shows where the error is click the red button so it goes there

Application Overview metrics for historical mode

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Over a historical range an alert color reflects the worst-case value of the heuristic at any point in the historical range For example if at any time during a historical range the User heuristic for an agent was yellow but never red the Overview tab for that historical range is yellow

Whatrsquos Interesting eventsThe lower half of the Overview lists Whatrsquos Interesting events which Introscope generates automatically when the color of an alert changes to yellow or red In Live mode the previous 20 minutes of events appear If you double click the error in overview then it shows the error message in intresting events

Shows the time at wt time it changes And also the message

EM overviewYou can view a variety of metrics on the Enterprise Manager itself by selecting the EM node under Custom Metric Agent

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

The EJB (Enterprise Java Beans) overview shows statistics for Entity beans Session beans and Message Driven beans

Frontend overviewsOverviews for Frontend nodes show graphed application metrics and statistics related to transactions in the application

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Overviews for Backend nodes show graph views of database metrics and a table view of SQL below the node

The garbage collection (GC) heap overview shows heap use

The Instance Counts overview tab shows the classes instantiated on the JVM

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

The JTA overview tab data about JTA components

The LeakHunter overview shows statistics graphically and in a table Leak tabs appear for nodes under LeakHunter and show details of the leak and a graph of the number of collections over time

The Servlet overview shows a table of servlets in the node When you select a servlet the Investigator shows its statistics in a graph

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

The socket overview shows tables for client and server sockets and socket information for each port

The Struts Overview tab shows an overview of Struts components with a display of the average response time for all components

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

The Threads overview shows all active threads being processed through an agent

The Overview tab for the XML node displays metrics for XML components

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Search tabThe Search tab is available when you select a node in the Investigator tree that contains metrics It enables you to quickly find metrics

Traces tabThe Traces tab available when a resource or component is selected in the Investigator tree

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Errors tabThe Errors tab available when a resource or component is selected in the Investigator tree lists errors and error details for the selected item

Metric Count tabMany of the nodes in the Investigator tree now have a new Metric Count tab which displays a pie chart of the metric distribution for the node

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Frontend metricsThese metrics are listed for each frontend1048708 Average Response Time (ms) 1048708 Concurrent Invocations 1048708 Errors Per Interval 1048708 Responses Per IntervalThe Heuristics node shows the metric values related to the alerts displayed in the Overview tab when an agent is selected

The URLs node under a front end node shows these metrics for each URL group that is configured for the front end

The Called Backends node contains metrics that reflect the activity and performance of a backend for a particular URL group

Viewing metrics for Backends in the InvestigatorFor each database backend these metrics reflect the activity and performance of the backend across all applications it serves

To change an alert view1048708 Display an alert in the Investigator Preview pane and select Properties gt AlertViewAlert messages are triggered by an action associated with an alert status These alerts appear automatically You can also view alert messages by selecting Workstation gt Show Alert Messages

Dashboard alerts that show overall statusEach sample dashboard includes alerts that show the overall state of the environment and how key performance indicators are affecting the environment

The Overview dashboardThe Overview dashboard is designed for the Application Support team to monitor the key performance indicators of their applications across the entire monitored environment

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

The Problem Analysis dashboard includes graphs that help you locate the cause of a particular problem

Introscope Transaction Tracer reduces the time required to identify a problem component in a transaction enabling authorized users to trace the transaction activity at the component level Transaction Tracer can trace synchronous transactions that cross boundaries in the homogeneous application server environments that support this capability1048708 WebLogic Server 80 and later1048708 WebSphere 6x

To start a Transaction Trace session1 Select Workstation gt New Transaction Trace Session The New Transaction Trace Session window opens

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

In the Trace transactions section of the window specify the threshold execution time Select milliseconds or seconds from the drop-down listTo stop a Transaction Trace session1048708 Click Stop or1048708 Select Trace gt Stop Tracing SessionTo restart a Transaction Trace session1048708 Click Restart or1048708 Select Trace gt Restart Tracing SessionTo turn off the warnings about low-threshold execution time1 Select Workstation gt User Preferences2 Click the Transaction Tracer tab3 Check the Donrsquot warn when threshold is less than 1 second checkbox4 Click Apply1 Select Trace gt Show Traced AgentsThe Tracing Agent(s) dialog box appears

Using the Transaction Trace ViewerThe Transaction Trace Viewer shows trace information for transactions that meet the criteria you specified for the trace session

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Summary View shows metrics for the components in the selected transaction Metrics include the path number of calls the length of the call in milliseconds and the minimum average and maximum call times

Trace View shows the selected transaction in graphical format

1 Select Workstation gt Query Historical EventsThe Historical Query Viewer opens2 In the Query field enter a combination of1048708 the query option type to include all transaction events that match the specified type1048708 a query stringmdashto search for errors that contain or match a string If you donrsquot enter a query string all errors events are returned1048708 query optionsmdashto limit your search based on event parameters

typeerrorsnapshot typenormal typesampled typewhatsinteresting

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

It will output a Transaction Trace XML file to the local directory and the dates are formatted

yyyymmddHH24We would like to extract the error details for particular application(agent)I tried with CProgram FilesCA WilyIntroscope8020libgtjava -Dhost=xxxx -Duser=xxxxx -Dpassword=xxxx -jar CLWorkstationjar get historical events matching (typeerrorsnapshot) and (agentwb1)and time[2010062712 TO 2010062718]For these things we create xml file and that stores through trace(open through xml) that works fine

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Querying for similar eventsIn Introscope you can query for events that are similar to a selected event For example similar events might be events that all contain the same components (Servlet gt EJB gt SQL) with varying response times Introscope considers events similar if 60 of the strings within them (component names SQL tables names and so forth) overlapTo save Transaction Trace data to an XML file1 In the Transaction Trace Viewer select the Transaction Traces to save1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Click Save As3 You can open the file now or select a location to save the file into enter a filename and click Save

To open saved Transaction Trace data in an XML file1 Select Workstation gt Query Historical Events2 Select Trace gt Open Saved Events (XML)3 Select the XML file from the browser window and click Open

1 In the Transaction Trace Viewer select the Transaction Traces to export1048708 CTRL + click to select multiple Transaction Traces1048708 Edit gt Select All to select all Transaction Traces in the window2 Select Trace gt Export

Manage Module editor A Management Module contains a set of Introscope monitoring configuration information You configure Introscopes monitoring logic by using Management Modules and elements which organize metric data in the Workstation

Management Modules for each domain contain elements Elements are objects that contain and organize metric data with monitoring logic for presentation in the Workstation Introscope elements are

Metric groupings Alerts (which includes Simple and Summary Alerts) Actions (which includes Shell Command E-mail and Workstation actions) Dashboards Calculators

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Report templates SNMP collections

A Sample Management Module is included in the SuperDomain when you install Introscope This Sample Management Module contains pre-configured dashboards that include commonly used performance-monitoring logic You must create other Management Modules for other created domain

Triage Map Configuration Management ModuleAnother useful Management Module is the Triage Map Configuration Management Module This Management Module saves definitions of alerts and other objects you create directly from the application triage map You can also use this management module to do the followingCreate alert downtime schedules for triage map alerts

Define alert actions for use in triage map alerts

To create a new metric grouping from an existing metric

1 Right-click the metric and select New Metric Grouping from Metric ltNamegt from the menu 2 Accept the default name for the metric grouping 3 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

4 Click OK

The new metric grouping you created is highlighted in the Management Module Editor tree under the Management Module in which you saved itNote The metric grouping is active when it is created and cannot be de-activated

In the Settings pane for the metric grouping metric grouping Agent Expressions and Metric Expressions fields already contain the metric information

Note Filtering based on agent name uses the property introscopeagentperfmonagentExpression A valid agent expression would be

ProcessName|AgentName or |MyAgent

5 Select the Description tab to enter descriptive text and any important information about the metric groupings in the Description Text field This field should contain no more than 64 KB of data After it is applied it will be persisted in the management module jar

6 Select which Agent Expressions to use o Select Use Management Module Agent Expressions to use the Agent Expressions

defined for the Management Module o Select Use Metric Grouping Agent Expressions to use Agent Expressions defined for

this metric grouping 7 Click Apply

To create a new metric grouping from the Elements menu

1 From any Workstation window select Workstation gt Management Module Editor 2 In the Management Module Editor window select Elements gt New Metric Grouping 3 In the Name field enter a name for the metric grouping 4 Choose a Management Module to contain the metric grouping in one of these ways

o Select a Management Module from the drop-down list box o Click Choose select a Management Module from the list then click Choose again

5 Click OK

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

To create a new Management Module 1 From any Workstation window select Workstation gt New Management Module Editor 2 In the Management Module Editor window select Elements gt New Management Module

3 Enter a name for the Management Module in the Name field (this name appears in the Management Module Editor tree) 4 Enter a jar file name for the Management Module using alphanumeric characters without spaces (to comply with all operating systems)

5 In the Domain Name field use the pull-down menu to select which domain contains the Management Module

6 Click OK

The Management Module appears in the Management Module Editor tree Modules are active and editable when they are created

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

In the Investigator click and hold a metric in the tree Drag it to the dashboard and drop it when you see the blue highlighted line all around the dashboard The illustration below shows what it looks like when you drag a metric from the Browse tab tree into a dashboard

Creating an empty data viewer To create an empty data viewer 1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select a dashboard object creation tool from the Tools Palette

3 Click and drag the location and size of the empty data viewer on the dashboard area 4 Release the mouse button The data viewer appears on the dashboard as drawn You can now manipulate the data viewer 5 Select File gt Save to save dashboard changes

Adding data to a data viewer by dragging and dropping You can add data to a data viewer by dragging and dropping from the Investigator or the Management Module Editor You can use this function to Add data to an empty data viewer

Replace the data displayed in existing data viewer

To add data to a data viewer by dragging and dropping

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

1 In the Console open the dashboard to edit by selecting Dashboard gt Edit dashboard 2 Select the data to add to the dashboard

To add a metric to the dashboard open an Investigator window and position it so both the Investigator window and dashboard are visible then click and hold a metric in the tree

To add an element to the dashboard open the Management Module Editor and click and hold an element in the tree

Creating report templatesIn the Management Module Editor select Elements gt New Report Templateraquo Note The New Report Template menu item is disabled if you do not have writepermissionThe New Report Template dialog box opens

a Type the Name for the new report templateb Select Force Uniqueness to ensure that the report name is unique If you select this option and you then enter a name that is not unique Introscope adds a number to the name to make it uniqueraquo Note The appended number appears after the report template is createdwhen you view it in the Management Module Editor If you donrsquot selectForce Uniqueness and an identical report template name existsIntroscope displays an error message and does not create the reportc Select a Management Module from the drop-down list box to choose the Management Module that will contain the reportd Optional Instead of selecting an existing Management Module to contain the report click Choose then click New Management Module and assign a name to the new Management Module The illustration below shows these options

For more information about creating Management Modules see the Introscope Administration and Configuration GuideThe new report template is added to the Management Module Editor and the settings pane opens

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

In the settings pane select the Active check box if you are ready to activate the report templateClick Open Template Editor to define report data

Adding report elements to reports1 If the report template editor is not already open open ita With the Management Module Editor open select the report in the pane on the leftb Click Open Template Editor2 Right-click the Report listed in the upper left pane and choose Add A list of available elements appears

A calendar dialog appears with the current date (ldquoTodayrdquo) circled Use the calendar dialog to set the date and edit the clock time in the text field after the dialog is closed

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Management modules ndash application ndash Metric groups here we can check all the things

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

So here we are putting the thresholds and if it cross its sending tickets and mails

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

We can change the mail names

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

To run the Introscope Enterprise Manager control scripts on UNIX Use the following commands at the bin directory EMCtrlsh startmdashThis starts the Enterprise Manager

EMCtrlsh stopmdashThis stops the Enterprise Manager

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

EMCtrlsh statusmdashThis gives the status of the Enterprise Manager

EMCtrlsh helpmdashThis shows the supported options for the script Gurusekhar(24021 AM) Do you know the introscope process name in systems Som(24227 AM) it runs as a startup class on managed server process Som(24232 AM) no seperate process

Introscope process starts along with the weblogic servers its add as one argument to that241 INTROSCOPE CONFIGURATION 242 WILY_HOME=optBEAweblogic103wlserver_103wilyAgentwily 243 WILY_BOOTPATH=-javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 244 WILY_OPTIONS=-DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 245 WILY_CLASSPATH=optBEAweblogic103wlserver_103wilyAgentwilyWebAppSupportjaroptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 246 CLASSPATH=$CLASSPATH$WILY_CLASSPATH 247 arguments=$arguments -DcomwilyintroscopeagentProfile=optBEAweblogic103wlserver_103wilyAgentwilyIntroscopeAgentprofile 248 arguments=$arguments -javaagentoptBEAweblogic103wlserver_103wilyAgentwilyAgentjar 249 INTROSCOPE CONFIGURATION End su $PRE_USER -c umask $UMASKexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH$JAVA $arguments $JAVA_OPTIONS weblogicServer 1gtgt$stdout_file 2gtgt$stderr_file amp

$ cd optBEAweblogic103wlserver_103$ ls -lrttotal 88190drwxr-xr-x 2 root other 96 Sep 1 2010 L10Ndrwxr-xr-x 9 root other 1024 Sep 1 2010 commondrwxr-xr-x 11 root other 1024 Sep 1 2010 serverdrwxr-xr-x 3 root other 96 Sep 1 2010 inventorydrwxr-xr-x 2 root other 96 Sep 1 2010 uninstalldrwxrwxr-x 3 eia eia 96 Dec 10 2012 wilyAgent-rw-r----- 1 root root 45117952 Dec 14 2012 wilyAgenttardrwxr-xr-x 2 sc744q eia 96 Dec 14 2012 introscope-rwxr-xr-x 1 root root 31882 Dec 14 2012 wlserver_176sh

$ pwdoptBEAweblogic103wlserver_103wilyAgentwilylogs$ view IntroscopeAgent20131104-230606log

-rw-rw-r-- 1 dyncp3 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213326log-rw-rw-r-- 1 dyncp2 dynamo 35637 Nov 4 2035 IntroscopeAgent20131104-213320log-rw-rw-r-- 1 wlldap weblogic 1640223 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt1log-rwxrwxrwx 1 wlmsgctr weblogic 1680 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr1log-rwxrwxrwx 1 wllogin weblogic 1538 Nov 4 2041 IntroscopeAgentwllogin__wllogin1log

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

-rwxrwxrwx 1 wlprfadm weblogic 1681 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm1log-rwxrwxrwx 1 wllogin weblogic 1539 Nov 4 2041 IntroscopeAgentwllogin__wllogin2log-rwxrwxrwx 1 wlprfadm weblogic 1619 Nov 4 2041 IntroscopeAgentwlprfadm__wlprfadm0log-rwxrwxrwx 1 wllogin weblogic 1666 Nov 4 2041 IntroscopeAgentwllogin__wllogin0log-rwxrwxrwx 1 wlmsgctr weblogic 1555 Nov 4 2041 IntroscopeAgentwlmsgctr__wlmsgctr0log-rwxrwxrwx 1 wlslfreg weblogic 1619 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg0log-rwxrwxrwx 1 wlsesmgt weblogic 1556 Nov 4 2041 IntroscopeAgentwlsesmgt__wlsesmgt0log-rwxrwxrwx 1 wlslfreg weblogic 1681 Nov 4 2041 IntroscopeAgentwlslfreg__wlslfreg1log-rwxrwxrwx 1 dyncp8 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp18log-rwxrwxrwx 1 dyncp7 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp17log-rwxrwxrwx 1 dyncp4 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp14log-rwxrwxrwx 1 dyncp5 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp15log-rwxrwxrwx 1 dynco1 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco11log-rwxrwxrwx 1 dyncp9 dynamo 2993 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp19log-rwxrwxrwx 1 dyncp2 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp12log-rwxrwxrwx 1 dynco3 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco13log-rwxrwxrwx 1 dyncp3 dynamo 1698 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp13log-rwxrwxrwx 1 dynco2 dynamo 1699 Nov 4 2045 IntroscopeAgentcommon_Common_cluster_dynco12log-rwxrwxrwx 1 dyncp6 dynamo 1572 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp16log-rwxrwxrwx 1 dyncp1 dynamo 1635 Nov 4 2045 IntroscopeAgentcp_cp_cluster_dyncp11log-rw-rw-r-- 1 mspdyn3 msp 116858 Nov 4 2048 IntroscopeAgentWebLogic_Agentlog-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2105 IntroscopeAgent20131104-220350log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2136 IntroscopeAgent20131104-223504log-rw-rw-r-- 1 wlldap weblogic 34580 Nov 4 2207 IntroscopeAgent20131104-230606log

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

$ pwd ls -lrtoptBEAweblogic103wlserver_103wilyAgentwilytotal 13490drwxrwxr-x 2 eia eia 96 Mar 25 2011 hotdeploy-rwxrwxr-x 1 eia eia 2088564 Mar 25 2011 WebAppSupportjar-rwxrwxr-x 1 eia eia 3457056 Mar 25 2011 Agentjar-rwxrwxr-x 1 eia eia 35423 Mar 25 2011 MQAppSupportjar-rwxrwxr-x 1 eia eia 10064 Mar 25 2011 ChangeDetector-configxml-rwxrwxr-x 1 eia eia 140926 Mar 25 2011 AppMapjar-rwxrwxr-x 1 eia eia 21022 Mar 25 2011 ChangeDetector-config-WebLogicxmldrwxrwxr-x 8 eia eia 1024 Mar 25 2011 examples-rwxrwxr-x 1 eia eia 7872 Mar 25 2011 libcdnativefileso-rwxrwxr-x 1 eia eia 3394 Mar 25 2011 jvmpbd-rwxrwxr-x 1 eia eia 25156 Mar 25 2011 jsfpbd-rwxrwxr-x 1 eia eia 800 Mar 25 2011 jsf-toggles-typicalpbd-rwxrwxr-x 1 eia eia 889 Mar 25 2011 jsf-toggles-fullpbd-rwxrwxr-x 1 eia eia 29394 Mar 25 2011 java2pbd-rwxrwxr-x 1 eia eia 22097 Mar 25 2011 j2eepbd-rwxrwxr-x 1 eia eia 3796 Mar 25 2011 errorspbd-rwxrwxr-x 1 eia eia 713 Mar 25 2011 dipbd-rwxrwxr-x 1 eia eia 3442 Mar 25 2011 bizrecordingpbd-rwxrwxr-x 1 eia eia 1997 Mar 25 2011 biz-trx-httppbd-rwxrwxr-x 1 eia eia 29972 Mar 25 2011 appmappbd-rwxrwxr-x 1 eia eia 49177 Mar 25 2011 appmap-soapbd-rwxrwxr-x 1 eia eia 19767 Mar 25 2011 appmap-ejbpbd-rwxrwxr-x 1 eia eia 6070 Mar 25 2011 WS_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 1743 Mar 25 2011 ServletHeaderDecoratorpbd-rwxrwxr-x 1 eia eia 3882 Mar 25 2011 PPWebLogicJMXFilterStringtxt-rwxrwxr-x 1 eia eia 28948 Mar 25 2011 OSBpbd-rwxrwxr-x 1 eia eia 1218 Mar 25 2011 OSB-typicalpbl-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-typicalpbd-rwxrwxr-x 1 eia eia 4922 Mar 25 2011 OSB-toggles-fullpbd-rwxrwxr-x 1 eia eia 1209 Mar 25 2011 OSB-fullpbl-rwxrwxr-x 1 eia eia 6979 Mar 25 2011 MQAgentproperties-rwxrwxr-x 1 eia eia 729 Mar 25 2011 CTG_ECI_Tracer_For_SYSVIEWpbd-rwxrwxr-x 1 eia eia 20612 Mar 25 2011 wl90xpbd-rwxrwxr-x 1 eia eia 21695 Mar 25 2011 wl100xpbd-rwxrwxr-x 1 eia eia 3323 Mar 25 2011 webspheremq_instancecountspbd-rwxrwxr-x 1 eia eia 108814 Mar 25 2011 webspheremq_errorspbd-rwxrwxr-x 1 eia eia 1807 Mar 25 2011 webspheremqpbl-rwxrwxr-x 1 eia eia 134433 Mar 25 2011 webspheremqpbd-rwxrwxr-x 1 eia eia 11758 Mar 25 2011 webspheremq-extrapbd-rwxrwxr-x 1 eia eia 168744 Mar 25 2011 webservicespbddrwxrwxr-x 2 eia eia 1024 Mar 25 2011 version-rwxrwxr-x 1 eia eia 4745 Mar 25 2011 taglibspbd-rwxrwxr-x 1 eia eia 2733 Mar 25 2011 summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2107 Mar 25 2011 strutspbd-rwxrwxr-x 1 eia eia 32611 Mar 25 2011 sqlagent-61pbd-rwxrwxr-x 1 eia eia 3499 Mar 25 2011 sql-agent-summary-metrics-61pbd-rwxrwxr-x 1 eia eia 2373 Mar 25 2011 soaagentpbd-rwxrwxr-x 1 eia eia 6898 Mar 25 2011 smweblogicasapbd-rwxrwxr-x 1 eia eia 2278 Mar 25 2011 smwebagentextpbd-rwxrwxr-x 1 eia eia 17531 Mar 25 2011 ppweblogicpbd-rwxrwxr-x 1 eia eia 6739 Mar 25 2011 powerpackforweblogicportal_typicalpbd-rwxrwxr-x 1 eia eia 6845 Mar 25 2011 powerpackforweblogicportal_fullpbd-rwxrwxr-x 1 eia eia 14621 Mar 25 2011 powerpackforweblogicportal_errorspbd-rwxrwxr-x 1 eia eia 2296 Mar 25 2011 powerpackforweblogicportalpbl-rwxrwxr-x 1 eia eia 37842 Mar 25 2011 powerpackforweblogicportalpbd

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

-rwxrwxr-x 1 eia eia 1867 Mar 25 2011 oraclejdbcpbd-rwxrwxr-x 1 eia eia 3494 Mar 25 2011 leakhunterpbddrwxrwxr-x 2 eia eia 96 Apr 26 2011 connectorsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 toolsdrwxrwxr-x 2 eia eia 96 Apr 26 2011 dynamicdrwxrwxr-x 2 eia eia 2048 Jun 20 2011 ext-rwxrwxr-x 1 eia eia 30276 Aug 9 2011 sqlagent-fullpbd-rwxrwxr-x 1 eia eia 30186 Aug 9 2011 sqlagentpbd-rwxrwxr-x 1 eia eia 6072 Sep 20 2011 toggles-typicalpbd-rwxrwxr-x 1 eia eia 2107 May 29 2012 weblogic-fullpbl-rwxrwxr-x 1 eia eia 6034 Jul 11 2012 toggles-fullpbd-rwxrwxr-x 1 eia eia 2119 Aug 31 2012 weblogic-typicalpbl-rwxrwxr-x 1 eia eia 60694 Dec 5 2012 IntroscopeAgentprofile_168-rwxrwxr-x 1 eia eia 60694 Dec 6 2012 IntroscopeAgentprofile-rwxrwxr-x 1 eia eia 405 Jan 10 2013 purgeIntroscopeLogsshdrwxrwxr-x 2 eia eia 40960 Nov 4 2208 logs

Add the Introscope Startup Class in each of the above Weblogic consoles as specified below

Login to the Weblogic Console In the left pane expand Environment Click on Startup and Shutdown Class under Environment Click the Lock and Edit button Click new Check Startup Class and click next In the name field enter ldquoIntroscope Startup Classrdquo In the class name field enter

ldquocomwilyintroscopeapiweblogicIntroscopeStartupClassrdquo Click next Place a check beside the managed servers and then click finish Click on Activate Changes Button

245 ltstartup-classgt 246 ltnamegtIntroscope Startup Classltnamegt 247 lttargetgtCommondynco3dynco1dynco2dynco13dynco12dynco11lttargetgt 248 ltclass-namegtcomwilyintroscopeapiweblogicIntroscopeStartupClassltclass-namegt 249 ltstartup-classgtThe application triage map The application triage map presents a graphical visualization of the components that make up your application showing application health and errors This map is automatically generated from performance and analysis of Introscope metrics errors and events It presents applications in the business-centric terms that yoursquove defined The application triage map enables you to instantly grasp the structure of and relationships between the applications in your environment in a visual manner to help you identify and triage current and emerging problems

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Management module is container that holds introscope elementsEvery thing in management module stored in a jar filethe jar file contains XML file where the definition of management module and its elements We can reuse the management modules in another EM

Action specifies what occurs when alert crosses user defined thresholdAlertDowntimescheduleAlerts notification generates based on thresholdDashboardwhere we can see the console workstationMetric groupinglogical group of metrics treated as single elementReportTemplatedefinition of structure and data that displays when a report is run It gets their data for metric groupingSNMPsimplenetworkmanagement protocolis used by the SNMP agent to specify the metrics to include in the management information base(MIB)

An agent expression is filter that uses the default agents to be used in the management module()|()|() --- any agent on any hostEJB|Session|MyBean|servicestailed Method Count --- Match only EJB |session|Mybean|servicestalled method countCPUUtlization - match any metric beginning with utilization in the cpu category

RBC_Back end stall count backends|ctgclientjdbcwily)stallcountRbc_sql_query backends|(wily((|SQLQeury|() Averageresponse()

Elements ndash New Managemenet module - give the name and jar file in the domain field use the drop down to select the domain(super domain)

Management module with in the same domain must have unique name In new management module when force uniqueness selected it changes non unique names to unique

Management module and edits are saved as XML and stored in jar files These are maintained by EM and should not change directly

After you create the management module we have to define the parameters for the management

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

In the settings under configuration check the active box this global switch all functionality onYou configure the agent expression which is default set of agent to use with this management moduleclick add to define agent expression This jar file locates in EMconfig directory

Management module you can delete or copy between domains and between EMS also

Metric grouping is logical set of one or more metricsuses a filter to group metrics from multiple sources at the same time Individual metrics are uniquely identified with fully qualified metric names

Host|weblogic|default|servlets|executorservletaverage response time

A Metric name average responsestall coutn indicated by

A metric grouping expression filter in a regex that defines the metric included in metric grouping

AgentExpression Define the agent from which the metric grouping will retrieve data ||Agent1First the part of the host that runs processes to be monitored second the name of the particular java process in an application or the instance of the managed java application from which you want to collect data Third the agent responsible for collecting the data

AB Server|WAS_NODE|AB_RealTime on host Abserver was node and the AB_Real time jvmABServer| --- All nodes in ABSERVER host

MetricExpressionDefines the metrics to include in the metric groupingServlets|Average response time

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Here under servlets many servlets are there and for all it gets the average response time We can create metric grouping in investgate only

Give the name and management module now it appears in management module-Metric grouping

In metric grouping for management module change the settings

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

the metric grouping agent expression looks like here it gets the GC Heap for the s1_66 node in all the JVMS

For the management module expression we use the Enterprisemanager|CPUEM CPU Used()

Simplet alerts enable compare the current value of one or more metrics in a metric grouping to user defined thresholdMultiple alerts will enable you to display the worst conditions among all the simple alertsYou can create actions to notify you when an alert occur so that we can reactCreating alert blackout enables you to temporarly disable alerts during routine servicemaintenance or load testing

Alerts generate when metric value crosses user defined threshold It contains two thresholds one for danger-red and antoher is caution-yellow

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Metricgroupingdefines the metric groping you want to watch the alert

DB connection is the metric grouping for the alertResolution time period between points when the comparison made between the metric value and the threshold

Comparison operator greater than or less than the threshold Combination its for any or all if the threshold changes any of the JVM it triggers if you select all it triggers all the threshold changes only

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Action delay when ever severity changes etc hellip

Alerts white no input data Green- acceptable valuesyellow- minor problem Red- errors

Right click on the metric grouping and simple alertsname and management module to select then it shows in the alerts side Notify the alert when to triggerlike the above settings While selecting you can find one option notify by individual metric donrsquotrsquo select this it created notification script that parses alert message text and then makes decision to route based on some text

Creating summary alerts status of two or more simple alerts state is defined as the worst among the simple alerts monitor Includes time stamp name of the summary alert current state of summary alert Summary alerts donrsquot have the user define time period for checking the current state of underlying simple alert Summary alerts are viewable with alert state data viewer only Element ndash Alert- Summary alert

Summary alert displays in the dashboard at starting to show the system health statusAll Alerts locates in emlogsalertstxt

Actions multiple alerts can share actions Element-new snmp notification actionnew send smtp mail actionshell command actionconsole notification action Use the console notification action ---- It displays the alert notification to all workstation it connected it comes infrontof the console saying errors are there

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Shell command action give the shell as echo 1 gtgt alertlogtxt

Send smtp mail action

Send mails to the team when ever the alert generates in Alerts only we can assign alerts

Add you can mention any type of actions Action delay setting apply only when trigger notification options are set to each period while problem exists or when severity increases

Alert blackout to schedule the deactivaton and reactivation of alerts elment ndash Alert blackout give the name Here select simple schedule or cron schedule For simple schedule mention the time and cron schedule check the cron This can be done in element- Alert blackout These all stores in EMconfigdirectoryltblackout alertsgt ltblackout enabled=true crong schedule ldquo0 30 3 SUN duration in mins =120gtltalert-regx mgmt ----------gt

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Dashboard ndash on the dashboard of console click Dashboard - new dashboard Data viewers display elements mapped to data as metricsmetric groupingalerts and calculators Link provides navigation between dashboards

You can get the dash board editor you can see the graph string viewer txt viewer bar chate rectangle etc hellipConfigure pixel size also From management module go to the metric grouping and drag and drop the metric gropuing to the dashboard editor or

Right click in the dataviewer- select data option there you can see all the metric groupingsalerts calculator and metrics select any Right click dataviewer- scale pin at 0 to maximum ndash auto expand and in sortfilter you can select the show the top 10 metrics by excluding the metrics over there Resolution configure the update period of the viewer in data like 15 seconds it refreshes

In the data viewer tab you can create your own graph and righe click object link to any of the investigator or management module

Summary reports like response per interval in the back end errors per interval in backend worst performance sql querys etc hellip want to generate on request It shows the display of historical events just drag and drop those all things from metric grouping or metrics etc hellip for sql query you many slect metric data table also

Reports are generated through CLW or the work station it provides critical information about enterprise web

applicationElements ndash New Report template ndash name and open template editorThere you can see the toolbargraphs and button Other side cover pagedefault data report and default display properties Drag and drop the template Preview selected elements On cover page give any name for the default data properties start time end time period etc hellip Report page report signature locale settingslanguage usedDefault display parameters

In the report editor ndash right click data text display properties create your own and do the linkGenerate report- workstation ndashgenerate report Select the report and click generate preview and ok save report to pdf

If we want SQL Query graph in editor select sql query time graph click the display properties use the option like show volume on yellow line value label redline value and label

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Detect and verify the problem if the problem is reoccurring verify the reports Prioritize the problem isolate the problem document the problem and forward it to specialist

Four kind of problems consistent- progressive ndash periodic- sudden

Load ndash the problem due to more user load code issues configuration issues and backend issues

Metrics with increased value ndash Average response time Metrics with decrease value response per intervalGC Bytes in use stalle method concurrent invocation

Consistent problem not belongs to timeload dependent not helps after adding capcity not solved to restart the system also Affects for some function usage It may be the too many layers between components over use of external system backend and code issues

Many Layers like object translation performance between two other objects needing to communicate with each other having too many layers between components can cuase slow behaviorA SQL query result is put into xml enhance that using XSLT read by data management layer DTO and retrun to EJB these all are many layersTo overcome above one sql query returns set of database record packages to specialized designed object of arrays other structure data might be returned to comma delimited text or electronic data exchange If we have so many standards also program needs to translate like one standard to another and come back

Too many layers impacts Metrics with increase value as AVg resp time concurrent invocations and stall count Decreased value as Response per intervalThis is code problem so report to the developer

Over useExternal system using LDAP to frequently lookup credentialssending emails to many users instead of one message n address list This happens for using back end system frequentlyMetrics with incrased value Avg resp time concurrent invocation Metrics with decreased value Available threadsThis can be to the developer

Slow back end response changing any server settings might cause this issue slow query transactions Metrics increased Avg resp concurrent invocation and stall count Decreased Available threadsDeveloper or database

Code issues too many loops calling same methods deadlock redundant objects Metric increasedAvg resp time concurrent invocation cpu utilization and decreased values are available threads and response per intervalDeveloper report this

Progressive issues severity range is abroad cpu and memory use increasedproblem usually effects all functions and server can eventually look upThis can be due to garbage collector resource leaks and memory leaks as relationship with GC and CPU instance count in GC and non expirting session objectsGC reclaims which objects are using the app make heap size available to other If any JVM locked it can assign to otherFor all progressive problem check the GC you can see this investigator for all app CPU JVM etc hellip

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Resource Collector File IO sockets and Threads opened and never closed When two threads occurs each other then dead lock occurs this is code issues Metrics incrasedconcurrent invocations stalled methods threads under servlets-threadpool etchellip Metrics decreased value responsible per interval available threads cpu utilization etc hellipTake threaddump to know the error

Memory leaks objects are added to the data structure and never released session expired but that session not removed from collection objects If this not used then only it can be removed by garbage collector Here cpu will reach 100 percent Metrics increasedAvg Resp timebytes use cpu utilization and instance counts Metrics decreased Resp per interval

Relationship between GC and CPUGarbage collect more often with leak cpu utilization spike ever time garbage is collected The GC hep byte use metric grows progressively The cpu spike with each GC dependent

Instance Count increases during the leak decreases during the GC

Non expiring session objectssession not expire after certain time Contact dev

Sudden problems is instantaneous(not linear or progressive) happens with out warning(worked fine one minute but not in the text Is recurring(but irregular time) The application server can freeze up start producing errors act chaoticallyLive locks are like Runaway process that consumes resourcesloop statement that doesnot specify when repletion stops Metrics with increased valueAvg resptime stall coutn cpu utilization count Decreasedresponse intervalUnending retrieswhen component makes constant call to unavailable systemsis the most common source of hang-ups and freezesrequires backd end lookup Here the threads drops to zero stall count metrics high cpul utliztion goes down

Pig and python The available memory is consumed with very large result set the jvm crashes as the bytes in use metric equals the total bytes metric Out of memor errors are received Increased resp time gc bytes usestall count concurrent invocation

decreased resp per interval Large result set should broken up to small oneSudden slow backendNetwork issues connection with another user less database resources content with antoher process and poor database designMetrics increases are avg resp time stall count and concurrent invocations Decreased available threads

Dead lock also

Periodic problems time of day or load time it recovers after some timeOver load metricsavailable threads are consumed as response takes longer concurrent invocation metrics are higher at the controller servlet and response per interval and cpu utilization is increases

Under allocation application server configured issues Increased metricAvg resp timeconcurrent invocaton Decreased cpu utilztion available threads available connections

In sufficient code logic is under load like process only one request at a time etc

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Load backend poorly written SQL Stored procedure Database server settings and Maiframe settings

Network issue in network issues every thing is normal at our endCalculators are the first example generating the custom metrics which exist on EM ProbeBuilder file extension(pbd) files are associated with java and net agents and defined what to monitor inside enterprise application EPAgents exist on managed server these are monitor from outside of the enterprise applicationsretrieving information from external processes and logs such as the JVM or CLR

Calculators are components that generate new metric based on calculations performed on other metrics Its deployed on EM where metrics can be intercepted and used Management and java script calculator performs customizable calculations on specified input metrics to produce new metric there is tradeoff between the simplicity of management calcualtors and the java script calculators

Virutal agents are new agents that aggregrate metrics from clustered applications that span multiple jvms and therefore multiple agents

1 Management module calculators develpement timeshort flexibilitylimited scopenarrow

2 Java script calculators Development timelong flexibilitygreat scopenarrow

3 Virtual agents Development timeshortflexibilitylimitedscopewide

Mangement module calculator performs the sum(counterstal count and concurrent invocations interval countresponse per interval and errors per interval) and average operations

Metric grouping ndash New calendar from ndash give the name you can see this in calenders

Java script calculators provide the full functionality of java script language create new metrics on any agent resides the EMscriptsjs files samples can be EMexamplesscripts

Java script calculator functions execute() works with input metric data and produce output javascript resultset helper getAgentRegex() specifies which agents are inputs is called once at deploymentgetMetricRegex() specifies which metric are inputs is called once at deployment getFrequency() specifies how often in seconds(mostly 15 seconds) that the execute method will create is called once at deployment

Virtual agents across multiple agents into new virtual agent Suppose in cluster(all of nodes) we have 16 agents each response per interval is 2512 etc hellip then crate virtual agent for this its avg response is like total by 16

It applicable application deployed to clusterd application serversmultiple instance of same application deployed on different jvmsThe metrics for the referenced agents are aggregated so that metric can be viewed at the logical application level and not only for each instance of the application in particular jvm Its like in all jvms one jsp is there it averages the response from all

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

These virtual agents reside on EM or Manager of managers in EM Cluster

Virtual agent calculators can include agents assigned to other virtual agent calculatorscannrsquot include other virtual agent calculators canrsquot include other virtual agent calculators must be uniquely named

Virtualagent config file EMconfigagentclusterxml

ltagentclustername=Myvirtualagenet domain=superdomaingt host|process|agentname

Java agents can change metric data by altering configuring or customizing by using pdb and URL grouping

The agent obtain list of applicable pbd files from introscopeagentprofile file and parses their directives

Through probebuilderthe agent inserts probes report performance and execution data to the agent the agent collects the data and reports metrics over the network to the EM During the running of application the probe reports performance and execution data to the agent the agent collect this data and reports metrics over the network to EM

JVM(Introscope agent)-introscope agentprofile- we mentioned either pbl or pbds ndash in pbl we mentions pbds

41 Directives Files 42 43 ================ 44 This property specifies all the directives files that determine 45 how Introscope AutoProbe performs the instrumentation Specify 46 a single entry or a comma-delimited list of entries The list 47 may include any combination of

48 - directives (pbd) files 49 - directives list (pbl) files 50 - directories that will be scanned about once per minute for

51 pbd files Directives files placed in a listed directory 52 will be loaded automatically without any need to edit this

53 Agent profile If dynamic instrumentation is enabled the 54 directives will take effect immediately without an app reboot

55 Non-absolute names will be resolved relative to the location of 56 this properties file 57 IMPORTANT NOTE This is a required parameter and it MUST be set 58 to a valid value 59 - If the property is not specified or the values are invalid 60 the Introscope Agent will not run

61 - If the property is set to include a directory and invalid 62 directives files are placed in the directory AutoProbe 63 metrics will no longer be reported 64 - If the property is set to include a directory and loaded 65 directives files are removed from the directory AutoProbe

66 metrics will no longer be reported 67 You must restart the managed application before changes to this property 68 take effect However if the property includes one or more directories 69 and dynamic instrumentation is enabled the Introscope Agent will load 70 directives files from the specified directories without an app restart 71 as noted above

73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeploy

gcp4al25gr417a$ pwd

optedbappwilyAgentwily

gcp4al25gr417a$ ls -lrt pbl

-rwxr-xr-x 1 beaadmin asadmin 2112 Sep 6 2012 weblogic-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1218 Sep 6 2012 OSB-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2126 Sep 6 2012 weblogic-typicalpbl

-rwxr-xr-x 1 beaadmin asadmin 2296 Sep 6 2012 powerpackforweblogicportalpbl

-rwxr-xr-x 1 beaadmin asadmin 1209 Sep 6 2012 OSB-fullpbl

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

-rwxr-xr-x 1 beaadmin asadmin 1807 Sep 6 2012 webspheremqpbl

gcp4al25gr417a$ view weblogic-typicalpbl

1

2 Introscope ProbeBuilder List File

3

4 CA Wily Introscope(R) Version 90 Release 9062

5 Copyright (c) 2011 CA All Rights Reserved

6 Introscope(R) is a registered trademark of CA

7

8

9

10 This file contains a list of ProbeBuilder directives files

11 These files contain the ProbeBuilder directives used when

12 adding probes into BEA WebLogic Server 90x 92 10x

13 or 103x

14

15

16

17 List Title

18 ================

19

20 Title BEA WebLogic Server [Typical] (90x 92 10x or 103x)

21

22

23

24 List Description

25 ================

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

26

27 Description Default Typical Introscope settings plus support for BEA WebLogic Server versions 90x 92 10x or 10 3x

28

29

30 Directives Files

31 ================

32 One directives file name per line Relative names

33 are resolved against the location of this file

34

35 toggles-typicalpbd

36 jvmpbd

37 j2eepbd

38 java2pbd

39 strutspbd

40 taglibspbd

41 oraclejdbcpbd

42 sqlagent-typicalpbd

43 errorspbd

44 summary-metrics-61pbd

45 sql-agent-summary-metrics-61pbd

46 wl90xpbd

47 wl100xpbd

48 jsf-toggles-typicalpbd

49 jsfpbd

50 biz-trx-httppbd

51 appmappbd

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

52 appmap-ejbpbd

53 bizrecordingpbd

54 dipbd

55

56

57 Servlet Header Decorator

58

59 Enable this PBD for CEM integration It appends additional

60 information to the HTTP response that is then captured by CEM

61 ================

62

63 ServletHeaderDecoratorpbd

64

65

66

67 LeakHunter

68

69 Enable this PBD to help locate the source of potential memory leaks

70 This will incur overhead so enable LeakHunter only when necessary

71 See LeakHunter documentation for usage and configuration details

72 ================

73

74 leakhunterpbd

gcp4al25gr417a$ view CTG_ECI_Tracer_For_SYSVIEWpbd

1

2 IBM CTG Java Gateway Tracer

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

3

4 Version 9062

5 Build 115259

6 Copyright (c) 2011 CA All Rights Reserved

7 Introscope(R) is a registered trademark of CA

8

9

10 SetTracerClassMapping JavaGatewayTracing comwilypowerpacksysviewctgtracereciJavaGatewayFlowTracer comwilyint roscopeprobebuildervalidateResourceNameValidator

11

12 SetFlag JavaGatewayTracer

13 TurnOn JavaGatewayTracer

14

15 IdentifyClassAs comibmctgclientJavaGateway JavaGatewayTracer

16 TraceOneMethodWithParametersIfFlagged JavaGatewayTracer flow JavaGatewayTracing CTGTracer|JavaGateway|Flow|server |program

Here we can find lot of pbd files many for the normal like jvm jsp there you can see the normal classes to exclude or monitor And some for application specific like above

Two types of pbl application-fullpbl and applicaton-typical pbl(more visibility more overhead better for QA and development environments)

Default metric set is modified as lsquo73 introscopeautoprobedirectivesFile=weblogic-typicalpblhotdeployrsquopbd and pbl updates require to restart the jvm

Turnon key activate a tracer group in the toggles-fullpbd or togglestypicalpbd files

gcp4al25gr417a$ ls -lrt togg

-rwxr-xr-x 1 beaadmin asadmin 6072 Sep 6 2012 toggles-typicalpbd

Example Web application using number of sockets to write large files in a remote systembut we are not able to see this due to networksocket IO is disable in toggles-typicalpbd file So go and uncheck that like turn on socket tracing

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

toggles-typicalpbd

TurnOn EJB2StubTracing TurnOn SessionBeanTracing TurnOn EntityBeanTracing TurnOn MessageDrivenBeanTracing TurnOn EJB3StubTracing TurnOn SessionBean3Tracing TurnOn EntityBean3Tracing TurnOn MessageDrivenBean3Tracing TurnOn EJB3MethodLevelTracing TurnOn HTTPServletTracing TurnOn JTACommitTracing etc hellip

Suppose if we want reduce overhead on jma transport then do turnonjavamailtransportflag

Over view of URL groupshow metrics appear in front end metrics gateway configured in the introscopeagentprofile url group defines how metrics will be segmented according to http request parameters

URL Grouping Configuration

================

Configuration settings for Frontend naming By default all frontends

go into the Default group This is done so that invalid URLs (ie

those that would generate a 404 error) do not create unique one-time

metrics -- this can bloat the EMs memory To get more meaningful

metrics out of the Frontends|Apps|URLs tree set up URL groups that

are relevant to the deployment

Changes to this property take effect immediately and do not require the managed application to be restarted

introscopeagenturlgroupkeys=default ( so here in url we can see all the jsps)

introscopeagenturlgroupgroupdefaultpathprefix=

introscopeagenturlgroupgroupdefaultformat=Default

if we give the above as introscopeagenturlgroupkeys=accountcarhotel then you can see these 3 url groups

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

We can use dynamic url also by introscopeagenturlgroupgrouppagesfromat=path_delimited24

Custom pbds are user defined pbds Select the suitable components from the system install the new pbd file in introscope system and update the introscopeagentprofile and pbl files

Example the three custom components that participate in application are customer email daemon and XML processer Customer is simple data holding object associated with an order whether object should go or email-daemon provide confirmation of ordered email for shipmentB2B processer is strong it implements a use case the full fillment of orders from business partners such as car rental hotel reservation through xml or webservice

Pbd files identify the classes and methods to trace the type of metrics to capture how the metrics will be displayed in the workstation investigator Files are divided into tracer groups set of probes associated with group of classes tracer grups are organized logically and custom tracer group is user defined set of monitoring instructions

Pdb ndash servlet tracer group(1) jms tracergroup(2) B2B order tracergroup(3) There are four steps to create 1)declares tracer group associate classes with your tracer group identify the methods to trace and how to trace them Activate the tracer group

To declare trace group use the set flag directive in pbd file to declare tracer group name example setflagb2b processorTracing

Use the identifyclass ascomrbcB2BOrderProcessor B2BProcessorTracing

The tracer directive often have four parameters the tracer groupname the method name to trace tracer type to use destination where the resulting metric

TraceonemethodifflaggedB2B processerTracing process orders(name of the method)Blame point tracer(tracer to use contains avg resp tme resp per interval errors per interval etc hellip Custom|B2bOrder processor|processorders

In pbd turn onB2B processor tracing

Deploy using introscope agentprofile or pbl(pbd file updates)check the Auto probe log

For subclassidentify inherited asassociate all child classes

Identify matchclass

This examples show the updated version of the custom PBD file created earlier monitoring the direct subclass of B2B order processor class using the identify inherited as directive and tracing complex methodsonce you made changes to the pbd file has to restart the EM

SetflagB2B processor Tracing

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties

Identify class as comrbcB2BOrderProcessor B2Bprocessortracing

Identify inherited ascomrbcB2Borderprocessor B2BProcesstracing

TraceComplexMethodFlaggedB2B ProcessTracing BlamepointTracerDifferentInstancesrdquoCustom|classname|methodrdquo

Turnon B2BProcessTracing

We can skip classes and method also

There are two ways of setting the stall count threshold in introscope Per agent and per method

Permethod threshold can be define by using the stalled method tracer in the tracer group The peragent is defined in theintroscopeagentprofile introscopeagentstallsthresholdseconds=30

EPAgent is standalone java program running on JVMretrieves metric data from plugin like normal agent from probes(pbd)

EPAgent input metric into CA introscope that extend beyond java and net monitoring Agent looks java classes etc hellip but EPAgent looks like WLS logreader plugin(binary format) etc hellip EPAgent starts plugins starts

You can install the EPAgent by bundle or standalone EMconfigEPAgentproperties