event moniter service

8/20/2019 Event Moniter Service

1/76

Using the Event Monitoring Service

Manufacturing Part Number:B7612-90009

March 1999

© Copyright 1999 Hew lett -Pa ckard Company


2/76

2

Legal Notices

Th e in for m a t ion con ta i ned in th is d ocu m en t is sub ject to ch a n ge w i th ou t

notice.

H ewl ett-Packar d m akes no warr anty of any ki nd wi th r egard t o this

manu al, in clu di ng, but not l i mi ted to, the im pl i ed w arr ant i es of

mer chan tabil i t y and fitness for a par ti cul ar pur pose. Hewlett-Packardsha ll not be liable for errors cont a ined herein or direct, in direct, specia l,

incidental or consequential damages in connection with the furnishing,

performa nce, or use of th is ma teria l.

Copyright © 1999 Hewlett-Packard Company.

This document conta ins informat ion which is protected by copyr igh t . Al l

rights a re reserved. Reproduction, ada pta tion, or tra nslat ion w ithout

prior w ritt en permission is prohibited, except a s a llow ed under t hecopyright laws.

Corporate Offices:

Hewlett-Packard Co.

3000 Hanover St.

Palo Alto, CA 94304

U se, duplica tion or disclosure by the U.S. G overnm ent D epart ment of

Defense is subject to res t r ict ions as set for th in paragraph (b)(3)(i i) of theRigh t s in Technical D a t a a nd S oftw a re claus e in FAR 52.227-7013.

Rights for non-DOD U.S. Government Depart ments a nd Agencies ar e as

set fort h in FAR 52.227-19(c)(1,2).

U se of this m a nua l a nd fl exible disc(s), compa ct disc(s), or ta pe

cart ridge(s) supplied for t his pa ck is restr icted to t his product only.

Ad dit i on a l copies of th e pr ogra m s m a y be m a d e f or secu r it y a n d ba ck-u p

purposes only. Resa le of th e progra ms in t heir present form or w ith

a ltera tions, is expressly prohibited.

A copy of th e speci fi c w a r r a n ty ter m s a p pl ica b le to y ou r Hew let t -Pa c ka r d

pr od uct a n d r epla cem en t pa r t s ca n be ob ta in ed f rom y ou r loca l Sa l es a n d

Service Offi ce.


3/76

Contents

3

1. Understanding the Event Monitoring Service

U nderst a nding E MS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

EMS Configur a tion Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

E MS R esource Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

U nderst a nding Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

U sing EMS in a High Availability Environment . . . . . . . . . . . . . . . . . . .17

E MS w ith MC /ServiceGua rd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

E MS w ith IT/Operat ions or Net work Node Man a ger. . . . . . . . . . . . . .17

EMS wit h Your Choice of Syst em Mana gement Softwa re . . . . . . . . . .18

2. Installing and Using EMS

Ins ta lling EMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Removing E MS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

St a rting t he Event Monitoring Service . . . . . . . . . . . . . . . . . . . . . . . . . .23

3. Selecting Resources to Monitor

Selecting Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

View ing R esource Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4. Defining a Monitoring Request

St a rt ing a Monitoring Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33

Specifying When t o Send a n E vent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Det ermining t he Frequ ency of Event s . . . . . . . . . . . . . . . . . . . . . . . . . . .36

Set ting th e Polling I nt erva l. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37

Selecting P rotocols for S ending E vents . . . . . . . . . . . . . . . . . . . . . . . . . .38

opcmsg (ITO) Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

TCP a nd U DP Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39


4/76

4

Contents

SN MP Tra ps Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Em a il Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Console Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Sy slog Option. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Textlog Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Adding a Notifi cat ion Comment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5. EMS Operations

Copying Monitoring Requ ests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Modifying Monitoring Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Removing Monitoring Request s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Viewing Monitoring Request s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

6. Configuring MC/ServiceGuard Package Dependencies

7. Troubleshooting

EMS Directories a nd Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Logging a nd Tra cing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

EMS Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

H igh Ava ilability Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

EMS Tra cing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Performa nce Considera tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Syst em Performa nce Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Netw ork Performa nce Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Testing Monitor Request s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Testing D isk Monitor Request s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Testing C luster Monitor Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Testing Net w ork Monitor Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Testing S yst em Resource Monitor Requ ests . . . . . . . . . . . . . . . . . . . . 68


5/76

Contents

5

Ma king Su re Monitors a re Runn ing . . . . . . . . . . . . . . . . . . . . . . . . . . .69

Glossary


6/76

6

Contents


7/76

7

Printing History

This edition documents ma terial r elat ed to insta lling a nd confi guring

Event Monitoring S ervice.

The printing da te chan ges wh en a n ew edition is printed. (Minor

corrections and updates which are incorporated at reprint do not cause

th e d a te to ch a n ge. ) Th e pa r t n u mber is r ev ised w h en ex ten s ive tech n ica l

changes are incorporated.

New editions of this ma nua l will incorpora te a ll mat erial updat ed since

th e previous edit ion.

HP Printing Division:

High Performance Systems Division

Hewlett-Packard Co.

19111 Pruneridge Ave.

Cupertino, CA 95014

Table 1

Printing Date Part Number Edition

March 1999 B7612-90009 Edition 1


8/76

8


9/76

9

Preface

This gu ide describes how to use E vent Monitoring Service (EMS ) alone,

a nd in conjunction with high a vailabili ty softw a re such a s

MC/Ser viceGu a rd a nd ent erprise ma na gement products like IT/O. The

cha pters a re a s follows:

• U ndersta nding the E vent Monitoring S ervice

• Insta l ling and U sing EMS

• Selecting Resources to Monitor

• Defi ning a Monitoring Request

• EMS Opera tions

• Confi guring MC /ServiceGua rd P a ckage Dependencies

• Troubleshooting

RelatedPublications

The follow ing document s cont a in a dditiona l relat ed informa tion:

• U sing H igh Avai lab i l i ty Moni tors (HP Part Number B5736-90012)

• M anaging M C/ Serv iceGuard (HP Part Number B3936-90026)

• EM S Hardwar e Event M oni tor i ng User’s Guid e (HP Pa rt NumberB 6191-90011)

• EM S Har dw ar e M oni tors Refer ence M anual (HP Pa rt Number

B 6191-90012)

• http://docs.hp.com/hpux/ha Web site for information about

Hewlett-Packard’s high-availability technologies where you can find

documents

• www.software.hp.com Web site for designing a nd building a n E MS

monitor. Select H igh Ava ilability t hen E vent Monitoring S ervice

Developer’s Kit.

• P et er Weyga nt , Clusters for H igh Avai l abi l i ty: A Pri mer of HP-UX

Solut ions (IS B N 0-13-494758-4). H P P ress: P rent ice Ha ll, In c., 1996

• Configuri ng OPS Clu sters wi th M C/ L ockM anager (HP Pa rt Number

B 5158-90019)

• M anagin g Systems and Work groups (HP Part Number B2355-90157)


10/76

10

• Tom Ma dell, Di sk and F il e M anagement Tasks on H P-UX (ISB N

0-13-518861-X). H P P ress; P rent ice Ha ll, I nc., 1997

• HP OpenView I T/ Operat i ons Adm in istrators Task Gui de (P /N

B 4249-90003)

• M anaging H ighly Avai l able NFS (HP Part Number B5125-90001)

ProblemReporting If you have a ny problems wit h t he softwa re or documenta tion, plea secont a ct your local Hew lett-Pa cka rd S a les Offi ce or Customer Service

Center.


11/76

11

1 Understanding the EventMonitoring Service

The E vent Monitoring S ervice (EMS ) is a fra mew ork for resourcemonitoring. It provides fa cilities to monitor sys tem resources including


12/76

12 Chapter1

Understanding the Event Monitoring Service

confi guring, checking resource sta tus, a nd sending notificat ion wh ere

confi gured condit ions a re met.

This chapter describes the following:

• Understan ding EMS

• U sing EMS in a High Availability E nvironment


13/76

Chapter 1 13


Understanding EMS

Understanding EMS

E M S ca n be con fi g ur ed t o poll a loca l s ys t em or a p plica t i on r es ou rce a n d

send messa ges wh en events occur. It can a lso be confi gured t o listen for

event messa ges from a synchronous monitors. An event is a ny a ction or

sta te you want to know a bout. For example, you ma y wa nt t o be alerted

w h en a d is k fa i ls or w h en a v a ila b le fi l es ys t em s pa ce fa l ls b elow a cer t a in

level. EMS a llow s you to confi gure w ha t you consider an event for anymonitored syst em r esource.

EM S sen d s even t n ot i fi ca t i on to a w i de va r iet y of softw a r e u sin g m u lt i ple

protocols (opcmsg, S NMP, TCP, U DP, sys log, console, t extlog, ema il). For

exa m pl e, y ou ca n con fi gu r e EM S to sen d a m essa ge to M C /Ser viceGu a r d

a nd I T/Opera tions w hen a disk fa ils. These a pplica tions can th en use

tha t messa ge to trigger package failover and t o send a messa ge to a n

a dministra tor to fi x the disk.EMS opera tes w ith a registry fra mework, a collection of monitors, and a

confi gura tion interface tha t r uns under S AM (HP -U X System

Administra tion Mana ger). EMS sta rts a nd stops the monitors, stores

informa tion used by the monitors, a nd directs monitors where to send

events.

Fig ur e 1-1 illustra tes th ese EMS r elat ionships.

Figure 1-1 Event Monitoring Service Operations

Target, such asIT/Operations,MC/ServiceGuard

Registrar

Framework

System andapplication

MONITORS

API

Resource

dictionary

Client, such asEMS the SAM interface to monitors,MC/ServiceGuard package configuration

resources


14/76

14 Chapter1


Understanding EMS

E M S w or ks in a h ig h a v a ila b ilit y en vir on men t t o h elp y ou q u ick ly d et ect

a nd eliminat e single points of fa ilure, a nd to monitor resources tha t canbe used a s MC/ServiceGua rd pa cka ge dependencies. EMS can a lso be

u sed ou t sid e a h igh a va i la b ili t y en vi ron m en t for m on itor in g th e s t a tu s of

syst em r esources.

EMS Configuration Requirements

Th is b ook d ocu men t s t w o w a y s t o a cces s a n d con fi g ur e E M S . You ca n u se

either or both of the following met hods or use your own m onitor client :

• Configure monitor ing requests through the EMS in ter f ace in the

Resource Management area of SAM.

• Confi gure package dependencies in MC/ServiceGuard by:

— u sin g th e Pa c ka ge C on fi gu r a t ion in ter fa ce in th e Hi gh Ava ila b ili t y

Clust ers suba rea of SAM, or

— edit ing the package ASCII confi gurat ion fi le.

The follow ing a re prerequisites t o using EMS :

• Monitored disks have been confi gured using LVM (Logical Volume

Manager).

• Netw or k ca r d s h a ve been con fi gu r ed .

• M ou n ted fi l esy stems h a ve been cr ea ted.

The follow ing a re syst em requirement s:

• All hardw are you intend to monitor, such as disks and LAN cards,

ha ve been confi gured an d t ested prior t o confi guring E MS.

• EMS must be insta l led on an H P 9000 Series 700 or Series 800

syst em run ning H P -U X version 10.20 or la ter.

EMS Resource Classes

Th e EM S fr a m ew or k gr ou ps r esou r ces in to cla s ses i n a h ier a r ch y s im il a r

to tha t of a fi lesystem str ucture, although the resources ar e not a ctua lly

fi les and directories. Some of th e resource class es supplied wit h t his

version of EMS a re listed in Figu re 1-2.


15/76

Chapter 1 15


Understanding EMS

Figure 1-2 Event Monitoring Service Resource Class Hierarchy

The full pat h of a resource includes th e resource class hiera rchy a nd

inst a nce. An exa mple of a full resource pat h for th e physical volume

sta tus of the device /dev/dsk/c0t1d2 belonging to volume group

vgDataBase, is /vg/vgDataBase/pv_pvlink/status/c0t1d2.

Crea ting a monitoring request mea ns indentifying th e full pa thna me ofth e r esou r ces to be m on itor ed a n d specif yin g th e con d it ion s u n der w h ich

contains all package,

contains network contains all logical volume, disk and PVlink, and volume group

node, and clusterstatus

interface status

contains all jobqueue, user, andfilesystem status

/net

/cluster

/pv_summary

/lv_summary /lv

/pv_pvlink

/package

/localNode / status

/interfaces

/lan

/ status

/jobQueue15Min

/numUsers

/filesystem

/ availMB

/status

/ status

/vgName

/ package_status

/ status

/ copies

/jobQueue1Min

/jobQueue5Min

summary status

/system

/vg

/ LANname

/ clusterName

/ clusterName

/ package_name

/ deviceName / lvName

/ lvName

/ fsName

Bold Italics -Bold -

Instances, replaced with an actual nameResource instance

/ service_status

/ package_name

/ service_name

Resource Cla sses from

Loca tion in Resource Hiera rchy: /


16/76

16 Chapter1


Understanding EMS

a n event notifi cat ion w ill be sent out by t he monitor.

Understanding Monitors

M on itor s a r e a p plica t i on s w r i t t en to ga th er a n d r epor t in for m a t ion a bou t

specifi c resources on t he syst em. They use sy stem informa tion st ored in

places like /etc/lvmtab and t he MIB da ta base. When you make a

request to a monitor t he follow ing sequ ence of events occurs:

• The monitor polls the resource to collect informa tion

• The monitor sends a message to the f ramework

• The f ramework eva lua tes the da ta to determine i f an event has

occurred

• I f an event ha s occurred , the f ramework sends not ifica t ion in the

appropriate forma t

To obta in ad ditional informa tion about inst a lled monitors:

1. G o t o t he /etc/opt/resmon/dictionary d irectory.

Ea ch monitor registered wit h EMS ha s an informa tion fi le tha t is

stored in t his directory.

2. View t he monitor dict ionary fi le.

The fi le na me is descript ive of each m onitor. The fi le extension is

.dict. For exam ple, the mibmonitor dictiona ry fi lenam e is

mibmond.dict.

A sta nda rd AP I provides a method for a dding new monitors t o EMS.

To crea t e your own monit or, refer t o the Wri t in g M oni tors for the Event

M oni tori ng Ser vice (EM S) Developer’s Kit :

1. Go to the high ava i l ab il ity w eb si te:

http://www.software.hp.com .

2. From the web site select High Availability, then select Event

M onit ori ng Ser vi ce Devel oper s K it .


17/76

Chapter 1 17


Using EMS in a High Availability Environment

Using EMS in a High Availability

Environment

EMS can detect loss of redunda nt r esources and report informa tion

a bou t i t . I d en t ify in g a n d r epor t in g s in g le poi nt s of f a ilu r e h el ps m a i nt a i n

a proactive approach t o preventing the loss of da ta a nd a vailability.

B eca use EMS observes a system, a nd does not modify t he system, youuse it w ith a dditiona l softwa re tha t can t ake action. Some useful

combina tions a re described in th e follow ing sections.

EMS with MC/ServiceGuard

M C /Ser viceGu a r d ca n be con fi gu r ed w i th EM S to d eter m in e th e h ea l th of

some resources, such as disks, an d ma y fail packag es over based on tha t

informa tion. Confi gure EMS r equests for use with MC /ServiceGua rd

packages either thr ough the Cluster a rea for Pa ckage Confi gura tion in

SAM, or by editing the pa ckage confi gura tion ASC II fi le.

Create requests through E MS:

• To enable a redundant not ifica t ion sys tem

• To monitor events tha t a f fect h igh ava i lab i li ty

• To be a ler ted to the cause of a package fa i lover

MC/ServiceGua rd ma y a lread y be confi gured to monitor the healt h of

n od es, ser vi ces , a n d subnet s , a n d to m a ke f a ilover d ecis ion s ba sed on th e

sta tus of these resources.

EMS with IT/Operations or Network Node Manager

EMS can be confi gured t o send events t o OpenView I T/Operat ions a nd

Netw ork Node Man a ger. Refer t o th e EMS Developer’s Kit . It conta ins

templates w hich a re a vailable for downloading a nd include instructions

for confi guring in Netw ork Node Ma na ger a nd I T/Opera tions.

To help confi gur e your Net w ork Node Man a ger a nd I T/Opera t ions for

EMS, refer to the Wri t in g M oni tors for the Event M oni tor i ng Servi ce

(EMS) Developer’s Kit :

1. Go to the high ava i l ab il ity w eb s ite :


18/76

18 Chapter1


Using EMS in a High Availability Environment

http://www.software.hp.com .

2. From the web site select High Availability, then select Event

M onit ori ng Ser vi ce Devel oper s K it .

EMS with Your Choice of System Management

Software

EMS can be used with a ny system mana gement softwa re that uses

messa ges sent in a ny of these protocols:

• S NMP t ra ps

• TCP

• U DP messa ges

• E ma il

• Text messages sent to a specia l ized file, including syslog


19/76

19

2 Installing and Using EMS


• Insta l ling EMS

• Removing EMS


20/76

20 Chapter2

Installing and Using EMS

• St a rting t he Event Monitoring Service


21/76

Chapter 2 21


Installing EMS

Installing EMS

EMS checks t he resources for loca l syst ems only. Ins ta ll EMS on every

syst em you wish t o monitor.

NOTE EMS is most effective when inst a lled an d confi gured on a ll systems in

your environment. U pdat ed monitors ma y have new sta tus va lues tha tchange the meaning of your monitoring requests.

The EMS bundle (Part Number B7609BA) version A.03.00 contains the

follow ing fi le sets:

• E M S -C or e—E M S fr a m ew or k

• EM S-C on fi g—SAM in ter fa ce to EM S

To insta ll EMS :

• U se swinstall, or

• U s e t h e S oft w a r e M a na g em en t a r ea i n S AM

If you have ma ny syst ems, it m ay be ea sier t o insta ll over the netw ork

from a cent ra l locat ion:

1. Create a network depot according to the instruct ions in Managing

H P-UX Sof tware wi t h SD-UX .

2. rlogin or telnet to the remote host on wh ich you are inst a lling

EMS.

3. Insta l l over the network from the depot .

When monitors a re upda ted, or wh en you re-insta ll a monitor on top ofa n existing monitor, your request s a re reta ined. This is pa rt of the

functionality provided by t he persistence client.


22/76

22 Chapter2


Removing EMS

Removing EMS

To remove EM S, use swremove or th e Softw a re Mana gement products

under SAM. Note tha t the monitors a re persistent , tha t is, they a re

always automatically started if they are stopped. Therefore, it is likely

you w ill have wa rnings in your remova l log fi le tha t sa y:

Could not shut down process

or errors tha t sa y:

File /etc/opt/resomn/lbin/p_client could not be removed.

E v en if y ou s ee t h es e w a r n in gs, m on it or s a r e r em ov ed a n d a n y d ir t y fi l es

a re clea ned up on reboot.

CAUTION Wh en y ou r em ove E M S, a l l log fi l es a r e a l so r em oved . I f y ou w is h t o s a vean y of the EMS log fi les, rename t hem or store them off before y ou

remove EMS.


23/76

Chapter 2 23


Starting the Event Monitoring Service


To star t EMS :

1. St ar t t he graphical version of SAM. From your comma nd line, type:

sam

2. Double-click t he Resource Mana gement icon.3. Double-click on th e Event Monitoring Service icon.

The ma in screen, shown in Fig ur e 2-1, shows a ll requests confi gured

on tha t system.

If th ere are no requests a nd S entinel monitors are not insta lled:

• A m es sa g e d is pla y s:

Currently no resources are being monitored. Use the

action.

• The fi eld a rea of the ma in screen i s empty.

If Sent inel monitors a re insta lled th e screen is simply blank.

Figure 2-1 Event Monitoring Service Main Screen


24/76

24 Chapter2




25/76

25

3 Selecting Resources to Monitor


• Selecting Resources

• Viewing Resource Descriptions

Selecting Resources to Monitor


26/76

26 Chapter3


Selecting Resources

Selecting Resources

All resources a re divid ed int o cla sses. To select a resource to monit or:

1. Cl ick on t he Actions menu.

2. Select Add Monitoring Request

The top-level resource classes for all installed monitors are

dyna mically discovered a nd t hen listed a s shown in th e fi gure below.

S ome Hew lett -P a ckar d products, su ch as ATM Ada pter for H P /9000

S ervers, H P OTS 9000 or Su pport Tools Ma na ger (STM) for HP 9000

ha rdwa re monitoring, include their own monitors w ithin t heir

pr od uct h ier a r ch y. I f th ose pr od uct s a r e in s t a lled on th e sys tem , th en

th eir t op-level resource class es a lso appear here.

NOTE After insta l l a t ion some monitors , including the STM hardware moni tors ,must be enabled before th eir resource clas ses appea r in t he EMS Add or

Copy Monitoring R equest screen.

Similarly, top-level resource classes belonging to user-written

monitors, crea ted using the procedures descr ibed in Wr i t i ng M on i t ors

for the Event M on i tor ing Serv ice (EM S) , a r e d iscover ed a n d d ispla y edhere.

To obta in a dditiona l informa tion a bout an y pa rticular monitor:

• R ev iew th e m on itor d ict i on a r y fi l e:

a . From the command l ine, go to the

/etc/opt/resmon/dictionary d irectory.

When a monitor registers w ith E MS, informa tion a bout t he

monitor ca n be stored in fi les in th e .dict d irectory.

b . View t he monitor dict ionary fi le.

The fi le na me is descript ive of each m onitor. The fi le extension

is .dict. For exam ple, the mibmonitor dictiona ry fi lenam e is

mibmond.dict.

• R eview t he m a n pa ge.



27/76

Chapter 3 27


Selecting Resources

The ma n pa ge nam e is referred to in the dictiona ry fi le in t he

MONITOR entry wit h th e monitor’s na me.

• View t he resource class or instance descript ion through EMS.

The EMS Monitoring Request P a ra meters screen ha s a View

Resource Description button that displays additional

informa tion supplied by th e selected r esource. S ee Figu re 3-1

Figure 3-1 EMS Monitoring Request Parameters Screen

3. Double-click on a resource clas s.

When y ou monitor a resource, you actua lly monitor one or m ore



28/76

28 Chapter3


Selecting Resources

specifi c insta nces of th a t resource clas s. The resource inst a nces,

a ssociat ed wit h t he selected resource clas s, display in t he ResourceInst a nce fi eld. See Figure 3-2.

If t he resource clas s ha s subcla sses, those subclas ses displa y in th e

Resource Classes fi eld.

Asynchronous monitors a re event-driven, ra th er t ha n polled. They

genera te messages in a continuous stream a s events occur a nd

w ithout rega rd for rela tive importa nce. Therefore if th e resource

in s t a n ce is f or a n a sy n ch r on ou s m on itor, th e C u rr en t Va l ue fi e ld d oesnot a pply, so the fi eld display s n/a.

Figure 3-2 Add or Copy Monitoring Request Screen



29/76

Chapter 3 29

g

Selecting Resources

4. Select a specific insta nce or th e wildcar d (All Insta nces).

The (*) w ild ca r d is a con ven ien t w a y t o cr ea t e m a n y r eq u es t s a t on ce.

Most syst ems have more than one disk or netw ork card, a nd ma ny

ha ve severa l disks. To avoid ha ving t o crea te a monitor request for

each disk, select *(All Instances) in t he Resource Inst a nce box.

The *(All Instances) l isting is alw ays t he last i tem on the list .

See the fi gure above.

Wildca rds a re ava ilable only w hen a ll inst a nces of a su bclas s ar e of

the same resource type and there are multiple instances. Selectingthe w ildcard a pplies the m onitor to a ll the insta nces of tha t resource

type.

Wildca rds a re not ava ilable for r esource clas ses. So, for exa mple, a

wildcar d is available for the sta tus insta nces in the

/sys tem /fi lesyst em/a va ilMb subclas s, but no wildca rd a ppear s for t he

volume group subcla ss un der t he /vg res ource cla ss.

5. C lick OK .

The Monitoring Request Parameters screen displays. See Chapter 4,

“Defi ning a Monitoring Request.”



30/76

30 Chapter3

g

Viewing Resource Descriptions

Viewing Resource Descriptions

Resource class a nd resource inst a nce descriptions a re a va ilable for ea ch

resource.

To view a resource cla ss descript ion, click th e View R esource Cla ss

Description but ton from t he Add or Copy Monitoring R equest screen.

To view a resource inst a nce description, click eith er t he View ResourceInst a nce Description butt on from t he Add or Copy Monitoring Req uest

screen or t he View Resource Description but ton from t he Monitoring

Request Pa ra meters screen.


31/76

31

4 Defining a Monitoring Request


• St ar ting a Monitoring Request

• Specifying When t o Send a n E vent

Defining a Monitoring Request


32/76

32 Chapter4

• Determining t he Frequency of Events

• Sett ing the Polling Int erval

• Selecting P rotocols for S ending E vents

• Adding a Notifi cat ion C omment


S i M i i R


33/76

Chapter 4 33

Starting a Monitoring Request

Starting a Monitoring Request

After y ou ha ve selected a resource to monitor, you ca n specify w hen a nd

how to send event notifi cat ion on the Monitoring Request P ar a meters

screen, (Fig ur e 4-1). The follow ing s ect ions describe th e monitorin g

par a meters a nd provide exam ples of common a pplica tions.

Figure 4-1 Monitoring Request Parameters Screen


Specifying When to Send an Event


34/76

34 Chapter4



When you creat e a request , you specify th e conditions un der w hich you

receive an a lert. While the monitor ma y be polling disks every fi ve

minutes, for example, you may only wa nt to be alerted wh en something

ha ppens tha t requires your a tt ention. S pecify these conditions in the

Notify a r ea of t h e M on it or in g R eq u es t P a r a m et er s s cr een . H er e a r e t h e

terms under w hich you ca n be notified:

When value is ... You define the conditions under which you wish to be

notified for a particular resource. Choose an operator

(=, not equal, >, >=,


35/76

Chapter 4 35


To set a n event tr igger:

• Select f rom the l is ted opt ions in the Notify a rea (When value

is..., When value changes, or At each interval).

Asynchronous m onitors a re event-driven, ra th er t ha n polled. They

generate messa ges in a continuous stream as events occur a nd

wi thout regard for rela t ive impor tance. Therefore, i f the request i s for

a n a synchronous monitor, only t he When value is... option is

available.

NOTE U pd a ted m on it or s m a y h a ve n ew s ta t us va lu es t ha t ch a ng e t he m ea n in gof your monitoring requests, or generat e new a lerts. For exam ple,

a ssume you ha ve a r equest for notifi ca tion if sta tus > 3 for a resource

w it h a v a lu es r a n ge of 1 t h rou gh 7. You r eceiv e a l er t s ea ch t im e t h e v a lu e

eq ua ls 4, 5, 6, or 7. I f t he u pd a ted ver sion of t h e m on it or h a s a n ew s ta t us

va lue of 1 th rough 8, you a lso see alerts w hen t he resource equa ls 8.


Determining the Frequency of Events


36/76

36 Chapter4



If you select th e When value is... from t he list in t he Notify a rea ,

th e O pt i on s a r ea d ispl a ys th r ee ch oices . Sel ect on e or m or e of th ese th r ee

options:

Init ia l U se t his opt ion to est a blish a ba seline w hen monitoring

resources such as available filesystem space or system

loa d. I t can also be used to test w hether newlyrequested events a re being sent.

Repea t U se t his opt ion for urgent a ler t s. The Repea t opt ion

sends an a lert a t ea ch polling interval a s long a s the

notify condition is met. U se this option w ith cau tion;

there is a r isk of high CP U use or fi ll ing log fi les a nd

alert windows.

Ret urn U se t his opt ion t o t ra ck w hen emergency sit ua t ionsreturn t o normal.

To set t he frequ ency of th e tr igger:

• C l ick one or more but tons on the li st in the Options a rea (Initial,

Repeat, and Return).

Asynchronous monitors a re event-driven, ra th er t ha n polled. They

genera te messages in a continuous stream a s events occur a ndwi thout regard for rela t ive impor tance.Therefore i f the resource is an

a synchronous monitor, the va lues in the Options a rea defaul ts to

Repeat an d cannot be cha nged.

This Options a r ea d oes n ot d ispla y if y ou h a ve selected When value

changes or At each interval from t he list in t he Notify a rea . In

these cases t he options default t o Initial a nd cannot be changed.


Setting the Polling Interval


37/76

Chapter 4 37



The polling int erva l specifi es how often th e resource monitor checks t he

resource value. The polling interva l is the ma ximum a mount of ela psed

time before a monitor knows about a cha nge in sta tus for a pa rticula r

resource.

The shorter th e polling interva l, the m ore likely you a re t o have recent

da ta . However, depending on the monitor, a short polling interva l mayuse more CP U a nd syst em resources. You need to weigh the a dvant a ges

a nd disadva nta ges betw een being able to quickly respond t o events a nd

ma inta ining good syst em performa nce. Some considera tions include:

• The minimum polling interval depends on the monitor ’s a bil ity t o

process quickly. For most resource monitors the minimum is 30

seconds. Disk monitor requests ca n be a s short a s 10 seconds.

• MC/ServiceGua rd monitors resources every few seconds. You may

wa nt to use a short polling int erva l (30 seconds or less) w hen it is

critica l tha t you ma ke a q uick failover decision.

• You m ay w a nt a pollin g in ter va l of 5 m in ut es or so for m on it or in g less

critical resources.

• You may w ant to set a very long pol ling interval (4 hours) to monitor

failed disks tha t a re not essential t o the system, but w hich should be

replaced in the next few da ys.

Asynchronous monitors a re event-driven, ra th er th a n polled, th ey

generate messages in a continuous stream a s events occur a nd w ithout

rega rd for r elat ive import a nce.Therefore if t he resource is a n

asynchronous monitor, the Polling Interval field displays n/a.

To set th e Polling I nt erva l:

1. Speci fy the quant i ty of t ime in the numbered fi eld .

2. Select th e unit of time from the unit of measur e field list (seconds,

minutes, hours, day). The m a ximum va lue is one (1) da y.


Selecting Protocols for Sending Events


38/76

38 Chapter4

g g


Through the Notify via a r ea specif y th e pr otocol y ou w a n t th e m on itor

to u se to sen d even t s. Th e opt i on s a r e d escr ibed in th e f ol low i ng sect ion s.

opcmsg (ITO) Option

This option sends messages t o ITO a pplicat ions via t he opcmsg da emon.I T O per a t ion M a na g ed N od e S oft w a r e 3.x or 4. x m us t be in st a l led on t h e

resource server running HP-UX version 10.20, for this option to display.

This opt ion is not current ly ava i lab le on systems running HP -UX vers ion

11.0.

The I TO messa ge severity options a re:

• Map from value (th is is not a va ilable on a ll monitors)

• Critical

• Major

• Minor

• Warning

• Normal

A specifi ed severity oth er tha n Norma l is retur ned under t he follow ingconditions:

• The When value is . . . condit ion chan ges from FALS E t o

TRUE.

For example if a d isk is being monitored, you wa nt n otifi cat ion w hen

th e disk is down. The When value is condition is FALS E w hile the

disk is up a nd runn ing correctly. The condit ion is TRU E, mea ning

a ction needs to be ta ken, when t he disk is dow n or not opera ting

correctly.

• The When value changes wh en the return va lue does not ma tch

the previous retur n va lue

Cert a in opcmsg monitoring request s ca n ma p directly t o severity level.

Select Map from value option from t he list in t he Severity a rea . This

is not a vailable with a ll monitors.




39/76

Chapter 4 39

Within t he opcmsg, EMS sets t he follow ing fi elds,:

• I TO a pplica t i on g r ou p to EM S(HP )

• m essa ge gr oup t o H A

• object t o the full pa t h of the resource being monitored

S ee HP OpenView I T/ Operat i ons Adm in istrators Task Guid e (Pa rt

Num ber B 4249-90003) for m ore inform a tion.

Templat es for confi guring IT/Operat ions a nd N etw ork Node Ma na ger t odisplay monitored events can be found on the Hewlett-Packard High

Ava i lab i li ty public web page a t http://www.software.hp.com . C lick on

High Availability, then Event M onit ori ng Ser vi ce Devel oper ’s K i t .

To set t he opcmsg prot ocol for ITO:

1. Speci fy t he notifi ca t ion t ype from t he l is t in t he Noti fy area .

2. Select the opcmsg (ITO) option from the list in t he Notify a rea.3. Select the severity f rom the l is t in the Severity a rea , (Map from

value, Critical, Major, Minor, Warning, Normal).

TCP and UDP Options

Th is s en ds TC P or U D P en cod ed even ts t o t h e t a r get h os t n a me a n d por t

indicat ed for th a t requ est. Thus t he messa ge ca n be directed t o a

user-w ritt en socket progra m.

To set th e TCP or UD P conditions:

1. S elect th e TCP or UDP option, as a ppropriat e, from the list in t he

Notify via a rea .

2. Speci fy t he tar get host na me and the port in t heir respective fields.

SNMP Traps Option

Th is sen d s m essa ges to a p pl ica t i on s u si ng SNMP tr a p s, such a s Netw or k

Node Ma na ger. See HP OpenVi ew U sin g Netw ork N ode M anager (P /N

J 1169-90002) for more informa tion on confi guring SN MP tr a ps.

The follow ing tr a ps ar e used by EMS :

EMS_ENTERPRISE_OID "1.3.6.1.4.1.11.2.3.1.7"

EMS_NORMAL_OID "1.3.6.1.4.1.11.2.3.1.7.0.1"




40/76

40 Chapter4

- Normal Event

EMS_ABNORMAL_OID "1.3.6.1.4.1.11.2.3.1.7.0.2"- Problem Event

EMS_REBOOT_OID "1.3.6.1.4.1.11.2.3.1.7.0.3"

- Reboot restart Event

EMS_RESTART_OID "1.3.6.1.4.1.11.2.3.1.7.0.4"

- Restart Event

EMS_NORMAL_SEV_OID "1.3.6.1.4.1.11.2.3.1.7.0.5"

- Problem Event w/Normal Severity

EMS_WARNING_SEV_OID "1.3.6.1.4.1.11.2.3.1.7.0.6"

- Problem Event w/Warning Severity

EMS_MINOR_SEV_OID "1.3.6.1.4.1.11.2.3.1.7.0.7"

- Problem Event w/Minor Severity

EMS_MAJOR_SEV_OID "1.3.6.1.4.1.11.2.3.1.7.0.8"

- Problem Event w/Major Severity

EMS_CRITICAL_SEV_OID "1.3.6.1.4.1.11.2.3.1.7.0.9"

- Problem Event w/Critical Severity

The Severity area options for SNMP tra ps are:

• Map from value, (this is not a vailable with all monitors)

• Critical

• Major• Minor

• Warning

• Normal

A specifi ed severity oth er tha n Norma l is retur ned under t he follow ing

conditions:

• The When value is . . . condit ion changes f rom FALSE to TRUE

For example if a d isk is being monitored, you wa nt n otifi cat ion w hen

th e disk is down. The When value is condition is FALS E w hile the

disk is up a nd runn ing correctly. The condit ion is TRU E, mea ning

a ction needs to be ta ken, when t he disk is dow n or not opera ting

correctly.


the previous retur n va lue




41/76

Chapter 4 41

Certa in SNMP monitoring requests can ma p directly t o severity levels.

Select the Map from value option from the list in t he Severity a rea .To set t he SNMP tra p:

1. Speci fy the notifi ca t ion type from the l is t in the Notify a rea .

2. Select the SNMP tra p option from the l is t in the Notify via a rea .

3. Select the severity f rom the l is t in the Severity a rea , (Map from

value, Critical, Major, Minor, Warning, Normal).

Email Option

This sends event notificat ion t o the email a ddress indicat ed for t ha t

request.

To set for a n ema il notifi cat ion:

1. S elect th e Email option from the list in t he Notify via a rea .

2. Speci fy t he ful l email address in the Ema il Address fi eld .

Console Option

This sends event notifi cat ion t o the syst em console.

To set for a console notifi cat ion:

• S elect the Console option from the list in t he Notify via a rea .

Syslog Option

This sends event notifi cat ion t o th e system log.

An a bnorma l event message (error) is retu rned un der t he follow ing

conditions:

• The When value is . . . condit ion changes f rom FALSE to TRUE


the previous return value

For a n a bn or m a l even t , a sys tem loggin g level of errorwi ll be associa ted

wit h t he logged message.

To set for a syst em log notifi cat ion:




42/76

42 Chapter4

• S elect t he Syslog option from the list in t he Notify via a rea .

Textlog Option

This sends event notificat ion t o the fi le indicat ed for t ha t r equest.

To set for a text log notifi cat ion:

1. S elect t h e Textlog option from t he list in t he Notify via a rea .

2. Speci fy the fi lename and path in the Fi le Pa th fi eld .

The default path, /var/opt/resmon/log/event.log, displays w hen

th e Textlog option is select ed.


Adding a Notification Comment


43/76

Chapter 4 43


The notifi cat ion comment is useful for sending ta sk reminders t o th e

recipient s of an event. For example, if you ha ve a disk m onitor request

tha t reports a n a ler t t ha t an entire mirror ha s fa i led, when tha t event

shows u p in IT/Operat ions, for exam ple, you may wa nt it t o have th e

na me of th e person to conta ct if disks fail.




44/76

44 Chapter4


45/76

45

5 EMS Operations


• Copying Monitoring Request s

• Modifying Monitoring Request s


46/76

46 Chapter5

• Removing Monitoring Request s

• Viewing Monitoring Request s


47/76

Chapter 5 47

Copying Monitoring Requests

There a re tw o way s to use th e copy function:

• To cr ea te r eq u es t s f or mul t i p l e resources using t he same monitoring

parameters .

• To cr ea te req u es t s f or th e same resource using different monitoring

parameters .To crea te req uests for mu ltiple resources using t he sa me monitoring

parameters :

1. From the Event Monitoring Service main screen, select t he

monitoring request w hose par a meters y ou wish t o copy.

You need to ha ve confi gured a t leas t one similar request for a simila r

instance.

2. Select Actions menu: Copy option.

The Add or Copy Monitoring Request screen displays.

3. From the Add or Copy Monitoring Request screen, select a different

resource insta nce and click OK.

The Monitoring Request Parameters screen displays.

4. C lick OK in t he Monitoring Request P ar a meters screen.

A message displays indicat ing th e new request ha s been added. The

E vent Monitoring Service ma in screen displa ys.

To crea te req uests for t he sa me resource using different monitoring

parameters :


monitoring request w ith t he insta nce for w hich you wish to havemult iple monitoring request s.

You need to ha ve confi gured a t lea st one request for th e insta nce.

2. Select Actions menu: Copy option.

The Add or Copy Monitoring Request screen displays.

3. C lick OK in t he Add or Copy Monitoring R equest screen.



48/76

48 Chapter5

4. Modify the para meters a s desired in the Monitoring Request

Parameters screen.

5. C lick OK.

A message displays indicating the new request has been added. The

Event Monitoring S ervice main screen displa ys.


49/76

Chapter 5 49

Modifying Monitoring Requests

To chang e the monitoring par a meters of a requ est:


monitoring request w hose par a meters you wish t o modify.

2. Select Actions menu: Modify option.


3. Modify the para meters as desired in the Monitoring Request

Parameters screen.

4. C lick OK.

A messa ge display s indica ting t he request ha s been modifi ed. The

E vent Monitoring Service ma in screen displa ys.


50/76

50 Chapter5

Removing Monitoring Requests

The Remove Monitoring Requests functions w ith m ultiple requests a s

w ell as single request s. To remove monitoring requ ests:


monitoring request you wish t o remove.

To select contig uous mult iple req uest s, hold the Shift key a nd click.

To select individua l mult iple requests, h old t he Ctrl key a nd click.

2. Select Actions menu: Remove option.

A Confirmation screen displays.

3. C lick OK.

A messa ge displa ys indicat ing how ma ny req uest(s) ha ve been

removed. The E vent Monitoring S ervice ma in screen displays.

4. To s t a rt m on it or in g t h e r es ou rce a g a in y ou m us t r ecr ea t e t h e r eq u es t ,

either by copying a similar r equest for a sim ilar r esource or by

re-entering the da ta .


51/76

Chapter 5 51

Viewing Monitoring Requests

To view t he para meters for a m onitoring req uest:


monitoring request you wish t o view an d either:

• D ou ble-C lick , or

• S elect Act ion s m en u: View

The View Monitoring Request Pa ra meters s creen display s. The

para meters l isted here mat ch th e pa ra meters specified for th e

monitoring request.

2. To modify the para meters of this request, click the Modify

Monitoring Request option.The Monitoring Request P a ra meters

screen display s. P roceed as described in “Modifying Monitoring

Requests.”

3. To close the View Monitoring Req uest screen, click OK.


52/76

52 Chapter5


53/76

53

6 Configuring MC/ServiceGuardPackage Dependencies

This cha pter describes how t o use SAM to crea te pa cka ge resource

dependencies on E MS r esources. Tha t is how t o crea te a monitoring

t t b th E MS d t tif MC/S i G d


54/76

54 Chapter6

request to observe the E MS r esource and to notify MC/ServiceGua rd

w hen th a t resource rea ches a critica l user-defi ned level.

MC/ServiceGuard wi ll then fa i lover the package. The fol lowing are some

examples of how E MS migh t be used:

• I n a clust er w her e on e copy of da t a is sh a red bet w een t w o n odes (bot h

con fi gu red w it h E MS ), y ou m a y w a n t t o fa il a pa cka g e over w h en , for

exa m ple, t he L AN or S C S I h os t a d a pt er fa ils on t h e n od e r un nin g t h e

packag e. MC/ServiceGua rd compa res t he resource “U P ” va lues on

other confi gured nodes, a nd fa ils the package over to the node th a t

ha s th e correct resources ava ilable.

• In a c lus ter where each node has i t s own copy of da ta , you may wa nt

to failover a package t o a nother node for a ny nu mber of rea sons:

• host ada pter, bus, controller, or disk fa i lure

• unprotected data (the number of copies is reduced to one)

• degraded performance because one of the P V l inks has fa i led

This informat ion for crea ting r equests is a lso va lid for E MS sold wit h

other products (ATM, OTS, or STM hardware monitors , for example) and

for user-wri t ten monitors wri t ten according to developer specificat ions in

Wri t in g M oni tors for the Event M oni tor i ng Servi ce (EMS) .

NOTE Creat e the sa me requests on all nodes confi gured for a n

MC/ServiceGua rd packag e.

A packag e can depend on a ny resource monitored by E MS. To crea te

package dependencies:

1. H a l t t h e clu st er. I n clu de a for ce opt ion t o s t op a l l pa ck a ges, b y t y pin g:

cmhaltcl -f

To a d d a n EM S r esou r ce, th e M C /Ser vi ceGu a r d clu ster m u st be d ow n .

You ca n m odify existing EMS resources th rough MC /ServiceGua rd

while the cluster is running.

2. From your command l ine, s tar t SAM, by typing:

sam

3. Double-click the C lusters icon.

4 Double click th e High Availability Clusters icon


55/76

Chapter 6 55

4. Double-click th e High Availability Clusters icon.

5. Double-click on the P ackage Configur at ion icon.

The High Ava ilability Clust ers screen shows a ll requests current ly

confi gured on t ha t syst em. If there are no requests currently

confi gured, the fi eld a rea is empty. Figu re 6-1 shows the High

Ava ilability C lusters screen wit h t he Actions menu d isplayed.

Figure 6-1 High Availability Clusters Screen


56/76

56 Chapter6

6. F rom t h e Act ion s m en u, s elect eit h er t h e Create/Add a Package or

Modify Package Configuration option.

I f you select Create/Add a Package, a s cr een s im ila r t o Figure 6-2,

displays.

Figure 6-2 Create/Add a Package Screen

I f h l d f


57/76

Chapter 6 57

7. I f y ou h a ve n ot pr ev iou sly d on e s o, s pecify a Package Name and Node

a nd specify a Package SUBNET Address. Then click on SpecifyPackage Resource Dependencies... to add EMS resources a s

packag e dependencies. A screen similar to Figu re 6-3 displays.

Figure 6-3 Package Resource Dependencies Screen

8 T k k g d d t EMS li k Add


58/76

58 Chapter6

8. To make a pa ckage dependent on an EMS resource, click Add

Resource.... The Resources: fi eld lists a ll the insta lled resourcesdiscovered by MC /S erviceG ua rd. S ee Figu re 6-4.

Figure 6-4 Add Resources Screen

9. C lick th r ou gh th e Resource Classes a nd Resource Names to select

th e entity you wish t o monitor. Click OK. A Resource Parameters

screen, simila r t o Figu re 6-5 displays.

The example in Figure 6-5 shows the poss ib le va lues for pv_summary.

Different resources show different available UP values.

Figure 6-5 Resource Parameters Screen


59/76

Chapter 6 59

NOTE Ma ke sure you a lwa ys select UP in the Resources Up Values fi eld .MC/ServiceGua rd creat es an E MS request t ha t sends a n event if the

Resources Up Value fi eld is not equa l to the UP value.

I f you select UP, t h e pa cka g e fa ils over if t h e va lu e is a n yt h in g bu t UP.

I f you select UP a nd PVG-UP, t he pa cka g e fa ils over if t h e pv_summary

value is not equa l to UP or PVG_UP; in other w ords, if pv_summary is

SUSPECT or DOWN.

The polling interva l determines th e ma ximum a mount of elapsed

t im e b efor e t h e m on it or kn ow s a b ou t a ch a n ge in r es ou rce s t a t us. For

critica l resources, you may w a nt t o set a short polling interva l, such

a s 30 seconds, but t his could a dversely a ffect sy stem performa nce.

With longer polling interva ls you gain sy stem performa nce, but y ou

risk n ot detecting problems soon enough.


60/76

60 Chapter6

You ca n a l so a d d r esou r ces a s pa cka ge d epen d en cies by m od if yin g th epacka ge confi gura tion fi le. The default fi lena me is

/etc/cmcluster/pkg_name.ascii. See Manag ing

M C/ Serv iceGuard for d et a ils on h ow t o m od ify t h is fi l e. A exa m ple of

the synta x is:

RESOURCE_NAME /vg/vg01/pv_summary

RESOURCE_POLLING_INTERVAL 60

RESOURCE_UP_VALUE = UP

RESOURCE_UP_VALUE = PVG_UP

MC/ServiceGuard automat ica l ly d is t r ibutes th is in format ion to every

node in the cluster w hen the pa ckage confi gura tion fi le is a pplied t o

th e cluster confi gura tion (see the cmapplyconf comma nd).


61/76

61

7 Troubleshooting

This section gives hint s on testin g your monitoring request s, an d gives

y ou s om e in for ma t i on a b ou t log fi l es a n d m on it or beh a vior t h a t w ill h elp

you determine the cause of problems. For informat ion on fi xing problems

detected by monitors, see th e list of relat ed publicat ions in t he P reface.

Troubleshooting

This cha pter h a s t he follow ing sections:


62/76

62 Chapter7

• EMS Directories a nd F iles

• Logging a nd Tra cing

• Performance Considerations

• Testing Monitor Request s

Troubleshooting

EMS Directories and Files


63/76

Chapter 7 63

EMS Directories and FilesEMS fi les a re locat ed in /etc/opt/resmon a nd /opt/resmon. Ta ble 7-1

lists fi les a nd directories th a t might h elp you determine the cause of

some problems:

Table 7-1 EMS Directories and Files

/etc/opt/resmon/config

This file sets how often EMS checks that monitors are persistent

(have not died).

/etc/opt/resmon/dictionary

This directory contains resource dictionaries for the various

monitors. The disk monitor resources are listed in

diskmond.dict and the cluster, network, and system resource

monitors are in the mibmond.dict. If you were writing your own

monitor, the dictionary would go in this directory.

/etc/opt/resmon/lbin

This is the directory where all the monitor daemons live. Some

important daemons in the directory:

p_client restarts any failed monitors based on information in the

config file.

registrar passes monitoring requests to the correct monitors.

Troubleshooting

EMS Directories and Files

Table 7-1 EMS Directories and Files


64/76

64 Chapter7

/etc/opt/resmon/log

This is a directory of log files used by EMS:

• client.log stores calls made by clients, such as

MC/ServiceGuard or the SAM interface to EMS.

• api.log stores api calls made by monitors.

• registrar.log contains errors found when reading the

resource dictionary.

• emsagent.log is the SNMP subagent responsible for sending

EMS events through an SNMP trap.

/opt/resmon/resls

This command lists the latest polled status of the specified resource

on a specified system.

This can be used to discover and get details on resources available

for monitoring.

/opt/resmon/resdata

This is used to get additional information from EMS regarding

events or monitor restarts.

Troubleshooting

Logging and Tracing


65/76

Chapter 7 65

Logging and TracingU se logging for most t roubleshooting a ctivities. B y default th e monitors

log t o api.log. Logging t o /var/adm/syslog/syslog.log is ON by

default for the disk monitor an d OFF by default for the remaining

monitors. Tra cing sh ould only be used w hen inst ructed t o do so by HP

support personnel. This is not a va ilable wit h a ll monitors.

EMS Logging

As m en t ion ed in th e pr ev iou s sect i on , l og fi l es i n /etc/opt/resmon/log

/cont a in informa tion logged by t he monitors.

Look at the client.log if you seem to be ha ving a problem wit h t he

SAM, or a ny oth er client, int erfaces to E MS or MC /ServiceGua rd. With

th e d ef a u lt l evel of l oggin g, on ly a u d it a n d er r or m essa ges a r e logged . An

example of a n a udit message is:

User event occurred at Thu Jul 31 16:13:31 1997

Process ID: 10404 (client)

Log Level:Audit

+ /vg/vg00/lv/copies/* (8 instances)

If (


66/76

66 Chapter7

Ent ries in /var/adm/syslog/syslog.log ar e marked with t he monitor

daemon name, for example diskmond or fsmond, fol lowed by the resource

na me a nd logging da ta . Additions, deletions, notificat ions, a nd chan ges

in resource sta tes a re logged. Er rors explaining w hy a resource is not

a va ila b le for m on itor in g, or w h y th e m on itor ca n n ot a ccess a r esou r ce a r e

a lso logged there.

Look at the registrar.log if you are ha ving tr ouble fin ding resourcesth a t you suspect exist on your syst em. This log cont a ins a ny errors th a t

w er e en cou nt er ed w h en t r yin g t o r ea d t h e d ict ion a r y. I f a d ict ion a r y w a s

cor ru pt ed in a n y w a y, t he r eg is tr a r w ou ld n ot be a ble t o r ea d it , a n d E MS

would not be a ble to fi nd t he resources a ssociat ed with tha t dictionary.

EMS Tracing

Som e m on itor s pr ov id e t r a cin g w h i ch ca n be u sed f or d ebuggin g m on itorcode.

Use the -d option to tur n on tr a cing for EMS . Tra cing should only be

used at the request of your H P support personnel when t rying to

determine if there ma y be a problem wit h EMS . To tu rn on tra cing,

modify the .dict fi le in /etc/opt/resmon/dictionary and add -d t o

th e monitor you w ould like to tra ce:

MONITOR: /etc/opt/resmon/lbin/diskmond -l -d

Kill the monitor process. The monitor will a utoma tically rest a rt w ith

tr a cing ena bled. To speed up monitor rest a rt , use the resls comma nd

w ith t he top level of th e resource class a s a n a rgum ent, for exa mple,

resls /vg.

Tra cing is customar ily logged t o

/etc/opt/resmon/log/monitor_name.log. The monitor_name usually

ma tches the na me used for th e monitor in the dictiona ry fi le. Forexample, the MIBmonitor uses mibmond.dict a nd mibmond.log.

Troubleshooting

Performance Considerations


67/76

Chapter 7 67

Performance ConsiderationsMonitoring your system, is importa nt to ma inta in high a vailabili ty, but

monitoring consumes system resources. You must carefully consider your

per for ma n ce n eed s a g a in st y ou r n eed t o k now a s s oon a s pos sib le w h en a

failure threat ens ava ilabili ty.

System Performance Issues

The prima ry performa nce impact w ill be relat ed to the polling interva l

a nd t he num ber of resources being monitored. You need to ba lan ce your

need to quickly detect failures w ith y our need for syst em performa nce

a n d a d ju st th e n u mber of r esou r ces y ou m on itor a n d th e pol lin g in ter va l s

accordingly.

For example, pv_pvlink/status resources include th e pv_summary

resource. You may wa nt to creat e only one pv_summary monitoringrequest ra ther th a n monitoring both pv_summary a nd

pv_pvlink/status for all disks.

For polling interva ls, you ma y w an t to set a short interval, such as 30

secon d s, for r esou r ces th a t r eq u ir e q u ick r espon se a f t er f a ilu r e, a n d set a

lon ger polli ng in ter va l , such a s 5 m in u tes or m or e, f or a l l oth er r esou r ces.

Network Performance IssuesAlthough m onitoring is not likely t o affect n etw ork performa nce, you

ma y wa nt t o ma ke sure tha t only necessary messages a re being sent.

Ma ke sure your monitor request s ar e confi gured so you a re notifi ed only

for importa nt events.

Troubleshooting

Testing Monitor Requests


68/76

68 Chapter7

Testing Monitor RequestsTo tes t th a t even t s a r e bei ng sen t , u se th e I NI TI AL opt i on a va ila b le w i th

conditiona l notifi cat ion w hen creat ing a monitoring request . This option

sends notifi cat ion on sta rtup. Exa mine it t o make sure your request is

properly confi gured a nd showing up in the correct system m an a gement

tool.

An a lternat ive is to use the “At ea ch interva l” notifi ca tion t o test tha t

events a re being sent in t he correct sys tem ma na gement t ool. Once you

es t a b lish th a t even t s a r e bein g sen t pr oper ly, y ou ca n m od if y th e r eq u es t .

Testing Disk Monitor Requests

C on fi g ur in g t h e I N ITI AL opt ion m a y b e en ou gh . H ow ever, if y ou w a n t t o

test tha t events a re sent w hen a disk fa i ls , you may wa nt t o detach the

bus or power down t he stora ge devices a nd see if events a re sent t o th eproper a pplica tion, or if MC/ServiceGua rd fa ils over t he a ppropriat e

package. This is only recommended on clusters th a t a re off-line, and not

being a ccessed by users.

To test /vg/vgName/lv/copies a nd /vg/vgName/lv/status, use the

vgchange com m a nd t o d ea ct iv a t e a n d a ct iv a t e t h e v olu me g rou p a n d s ee

if the proper alerts w ere sent.

Testing Cluster Monitor Requests

Use the cmviewcl -v command to display detailed information about

th e current sta tus of the clust er a nd pa cka ges on t he cluster. The EMS

cluster monitor should return the sa me values a s th is comma nd.

Testing Network Monitor Requests

I f y ou w a n t t o t es t w h et h er ev en t s a r e s en t in ca s e of n et w or k fa ilu re, u se

the /usr/bin/ifconfig LANname down com ma n d t o br in g a ca r d d ow n ,

an d examine the event t o make sure it shows up in t he correct system

ma na gement t ool.

Testing System Resource Monitor Requests

Use the uptime comma nd t o verify the number of users a nd t he system

Troubleshooting


loa d. The EMS syst em resource monitor should return th e sam e values

tha t t his comma nd does.


69/76

Chapter 7 69

Making Sure Monitors are Running

Monitor daemons aut oma tically sta rt w hen you creat e a m onitoring

request. B ecause monitoring is designed to w ork in a high a vailabili ty

environment, monitors ar e writt en to aut oma tically resta rt i f anyt hing

ca uses them t o fail .

A daemon called p_client resta rts a ll appropriat e monitors using themonitor restart interval defin ed in /etc/opt/resmon/config.

Therefore, a monitor cann ot be perma nent ly stopped or st a rt ed by a

human.

Beca u se th e m on itor s a r e per s is t en t , m on itor in g r eq u es t s a r e kept w h en

you inst a ll a new m onitor or upda te a n existing monitor. If a condition,

s uch a s “ st a t us > 3” is bein g m on it or ed for a r es ou rces t h a t h a s a r a ng e of

1-7, and new version of monitor is installed tha t supports a new st a tus

value, such a s “8”, you ma y sta rt seeing notifi cations for “sta tus= 8”.

Troubleshooting



70/76

70 Chapter7

Glossary


71/76

71

A-H

alert An event. A messa ge sent t owa rn a user or a pplicat ion w hen

certain conditions are met.

asynchronous monitor Amonitor that monitors resource

inst a nces (or resource class)

a synchronously. It is event driven

a n d sen d n ot i fi ca t i on s w h en even t s

occur. It does not keep tr a ck of th e

current st a te or value of each

resource it monitors.

client The applica t ion tha t createsor ca ncels requ ests t o monitor

par ticular resources. The

consum er of a r esource sta tus

messa ge. A user of th e Resource

Monitor f ramework. This user may

browse resources, request status,

and make requests t o have

resources monitored. Examples are

MC/ServiceGua rd a s i t st a rts a

packag e or th e SAM interfa ce to

EMS.

dictionary See Resource

Dictionary.

EMS (Event Monitoring Service)The int erface betw een resource

monitors, the client a nd t a rget

applications.

EMS Framework A set of AP Istogether wit h the Registra r

process a nd t he resource

dictiona ry, w hich a llow client

a pplica tions t o request t ha t

resources be monitored a nd a

ta rget a pplicat ion be notified.

event An alert.

HA H igh Ava ilability.

I-L

ITO H P OpenView IT/Opera tions,formerly known as Opera tions

Center

logical extent The ba sica l loca t i on u n it f or a log ica l volu m e

is called a logical extent . For

mirrored logical volumes, either

tw o or th ree physica l extents a re

ma pped for each logica l extent,

depending on w hether you ar e

using 2-wa y or 3-wa y m irroring.

logical volume A collection ofdisk spa ce from one or more disk s.

E a ch collection a ppea rs t o theopera ting syst em as a single disk.

Like disks, logical volumes can be

u sed t o h old fi l e sy st em s, r a w d a ta

areas, dump areas, or swa p areas.

U nlike disks, logica l volumes ca n

be given a size when they a re

creat ed, an d a logica l volume ca n


72/76

72

l a ter be expanded or reduced. Also,

log ica l vol um es ca n be spr ea d over

mult iple disks.

LUN (Logica l U nit Numbers) Alogica l disk device composed of one

or more physica l d isk mechanisms,ty pica lly confi gured int o a R AID

level.

LVM (Logical Volume Manager)Ma na ges disks in volume groups,

a nd a llow s you to crea te logica l

a nd phys ica l volume groupings.

M

MIB (Man agement Informa tionB a se). A document th a t describes

objects to be ma na ged. A MIB is

created using a gramma r defined

in “Str ucture of Man agement

Informa tion” (SMI ) forma t. This

gra mma r concisely defi nes the

objects being ma na ged, the dat a

ty pes these objects ta ke,

descr ipt ions of how the objects can

be used, wheth er th e objects a re

rea d-only or rea d-w rite, a nd

identifi ers for the objects.

MIB II (MIB2) A MIB t hatdefines informa tion a bout t he

system, the netw ork interface

car ds it conta ins, routing

informa tion it cont a ins, the TCP

an d UD P sockets it conta ins and

their stat es, and various sta tistics

r ela ted to er r or cou n t s. Th is M IB is

widely a dopted a nd is served by

most I P -a ddressed devices. Most

system a nd n etwork resources

m a n a ged by E M S H A M on it or s a r e

ta ken from this MIB.

monitor S ee resource monitor.

N-P

notification See alert.

physical extent LVM divideseach physical disk int o a ddressed

units called physical extents.

physical volume A disk tha t ha sbeen initia lized by LVM.

polling The process by w hich amonitor obta ins th e most r ecent

sta tus of a resource.

PVG (physical volume group)

A grouping of phy sical d evices

(host ada pters , busses, controllers ,

or disks), th a t a llow LVM to

ma na ge redunda nt l inks ormirrored disks an d a ccess the

redundant ha rdwa re when the

primary hardw are fa i ls.


73/76

73

PV links A met hod of LVMconfi gura tion tha t a llows you to

provide redundant SCSI inter faces

a nd buses to disk arra ys, thereby

protecting a ga inst single points of

failure in SC SI car ds an d cables.

Q-R

registrar The r egistra r processprovides the l ink between resource

sta tus consumers (client s) a nd

resource sta tus providers (resource

monitors). The cent ra l par t of the

resource monitor framework which

uses th e resource dictiona ry t o act

a s a n intermediary between client

syst ems an d resource monitors.

resource May be any enti ty amonitor application developer

na mes. Exa mples include a

netw ork interfa ce, CP U st a tistics,

a MIB object, or a netw ork service.

resource class A cat egory ofresources useful dur ing

confi gura tion. For exam ple,

/net /in t er fa ces/la n/st a t us is

provided a s a resource class.

resource dictionary A fi ledescribing t he hiera rchy of

resources tha t can be monitored

a n d th e pr ocesses th a t per for m th e

resource monitoring.

resource instanceThe actua lresources tha t can be monitored.

For example,

/net /int erfa ces/la n/st a t us/la n0 ma y

refer to a pa rticular netw ork

interface installed on t he

monitored system.

resource monitor A frameworkfor selecting resources of interest

a nd monitoring t hem a ccording t o

th e user's criteria . When t he

resource value ma tches th e user's

criteria , a notificat ion is sent

a ccording t o the user'sinstructions.

The process th a t is used t o obta in

the st at us of a resource an d send

event notificat ions if a ppropriat e.

A monit or checks resources on t he

local system. The resource monitor

ma ps the phys ica l resource int o asta nda rd interfa ce understood by

EMS.

S-T

SNMP (Simple Netw orkMan agement P rotocol) St an dar d

protocol for net w ork ba sed

retr ieval of informa tion about

syst em resources.

state The curren t v a lue of aresource (UP or DOWN). For some

resource inst a nces, a monitor ma y


74/76

74

need to maint a in a hist ory of past

events or condit ions in order to

know th e resource va lue. In t his

case, a monitor is said t o be

maintaining state information.

S t a te less m on itor s d o n ot keep a n y

history of past conditions.

target The ta rget a pplica tion isnot ified when a moni tored resource

rea ches t he condition for w hich

notificat ion w a s requested. For

example, a ta rget appl ica t ion could

be MC/S erviceG ua rd or

IT/Oper a t ions (ITO).

U-Z

volume group In LVM, a set ofphysical volumes w hose extent s

ar e grouped together and then

ma de ava ilable to users as logical

volumes. A volume group ca n be

activat ed by only one node at a

time unless you a re using

MC /LockMa na ger.

MC/ServiceGua rd can a ctivat e a

volume group when it st ar ts a

package. A given disk can belong

to only one volume group. A logica l

volume can belong t o only one

volume group.

Index


75/76

Index 75

A

a pi.log fi le, 65asynchronous

monitors, 37a synchronous monitors, 28

C

classes , 28client.log file, 65cluster , 54cluster monitor request

test ing, 68console

notifi cation option , 41copying requests , 47creating a m onitoring request ,

33creating package dependencies,

54

D

dictionary, 63disk monitor request

test ing, 68

E

emailnotifi cation option , 41

E MSfi les and directories, 63logging, 65restart ing , 69SAM interface, 13, 23star t ing, 23testing monitoring requests,

68tracing, 66

EMS, high a vai labi li ty, 17event

notification , 34notifi cation frequency , 36

notification protocol options,38

opcmsg I TO option , 38SNMP notifi cat ion, 39

F

fi les and directories conta iningEMS monitors, 63

H

H Abasic, 13install ing , 21MC/Serv iceG ua rd , 17prerequisites, 14s yst em management s of t wa r e,

18hierarchy , 28high a vai labi li ty, 17

I

informationa bout monitors, 26

initial notification , 36install ing

over netw ork, 21instal l ing H A, 21IT/Oper a t ions , 39

send event , 17ITO

severity options, 38

L

log fi les, 63, 64logging, 65

M

MC /ServiceG ua rd , 17, 54creat ing package

dependencies, 54HA Monitors, 17

moidfying requ ests, 49monitor , 16

AP I , 16asynchronous, 28, 37fi nding informa tion , 26view informa tion, 16

monitor daemons, 63monitor persistence, 69monitoring request

copying , 47creating , 33crea ting comment s, 43modifying , 49polling interva l, 37removing , 50

test ing, 68viewing, 51monitors

updat ing, 35

N

Network Node Managersend event , 17

notification , 34event frequency , 36

notifi cation comment , 43notifi cation option

console, 41email, 41syslog, 41TCP , 39textlog , 42U D P , 39

notifi cation options, 36

notification protocol, 39SNMP , 39

Noti fy at each interval, 34Notify when value cha nges, 34Notify wh en va lue is. . ., 34

O

opcmsg

Index


76/76

76 Index

event notification , 38options for notifi cation, 36

P

p_client, 69package confi gurat ion fi le, 60package dependencies, 60

creating , 54performace, 67persistence, 35, 69polling interva l, 37, 60, 67protocol

event notifi cation options , 38protocol, sending notifi cat ion , 39

Rregistra r. log fi le, 66removing requ ests, 50repeat not ifi cat ion , 36resource

a s MC/Serv iceG ua rddependency, 54

selecting, 26UP va lues, 59

viewing descriptions, 30resource cla sses, 14, 28resource dictionary , 63resource monitor , 16restart ing EMS , 69return not ificat ion , 36

S

SAM interface

E MS , 13SAM interface to EMS , 23SAM interface to

MC/Serv iceG ua rd , 57selecting a resource, 26sending

events, 34severity options

ITO, 38

SNMP , 39SNMP

severity options, 39SNMP t raps, 39star t ing

E MS , 23

syslognotifi cation option, 41

system load, 67system requirements, 14

T

TCPnotifi cation option, 39

testing monitoring requests, 68

textlognotifi cation option, 42

tracing, 66

U

U D Pnotificat ion option, 39

UP v a lue, 59updating monitors, 35

V

viewingresource descriptions, 30

viewing requests, 51

W

wildcard, 29

event moniter service

Documents