OS* Toolsfor

Monitoring Progress Applications

*Including Windows even though Gus says that I am lying ;)

2

A Few Words about the Speaker

• Tom Bascom; Progress user & roaming DBA since 1987

• President, DBAppraise, LLC– Remote database management service for OpenEdge.– Simplifying the job of managing and monitoring the world’s

best business applications.– tom@dbappraise.com

• VP, White Star Software, LLC– Expert consulting services related to all aspects of Progress and

OpenEdge.– tom@wss.com

Scope

• OS oriented – not specific to Progress or OpenEdge.

• But useful with Progress and OpenEdge• Emphasis on Free tools.• Pre-installed or easily available.

Common Use Cases

• CPU Utilization• Runaway Processes• Memory Utilization

• Disk Space• Disk Activity• Disk Capacity• Progress Temp Files

CPU Utilization

• Am I really using all of my CPUs? (or “cores”?)• Am I getting useful work done?• Do I have room for growth?• Do I have a “runaway process”?

sar -u

sar -u 5 5

HP-UX pluto B.11.23 U ia64 11/11/10

15:00:00 %usr %sys %wio %idle15:00:05 5 10 20 6515:00:10 6 10 15 6915:00:15 5 10 25 6015:00:20 30 8 5 5715:00:25 24 2 5 69Average 14 8 14 64

sar -u

sar -u 5 5

HP-UX pluto B.11.23 U ia64 11/11/10

15:00:00 %usr %sys %wio %idle15:00:05 5 10 20 6515:00:10 6 10 15 6915:00:15 5 10 25 6015:00:20 30 8 5 5715:00:25 24 2 5 69Average 14 8 14 64

sar -f /usr/adm/sa/sa11 -s 15:00 -e 15:45 -i 600 -u

HP-UX pluto B.11.23 U ia64 11/11/10

15:00:00 %usr %sys %wio %idle15:10:00 80 5 10 515:20:00 82 6 8 415:30:00 83 4 13 015:40:00 82 5 12 1Average 82 5 11 3

topProcesses: 72 total, 2 running, 70 sleeping, 325 threads 15:33:28Load Avg: 0.22, 0.22, 0.17 CPU usage: 1.26% user, 2.10% sys, 96.62% idleSharedLibs: 8260K resident, 9820K data, 0B linkedit.MemRegions: 15320 total, 833M resident, 29M private, 676M shared.PhysMem: 919M wired, 1558M active, 588M inactive, 3064M used, 5128M free.VM: 154G vsize, 1042M framework vsize, 198231(0) pageins, 0(0) pageouts.Networks: packets: 2049591/276M in, 2286325/256M out.Disks: 89638/2507M read, 102460/3753M written.

PID COMMAND %CPU TIME #TH #WQ #POR #MREG RPRVT RSHRD RSIZE 2465 top 5.2 00:00.53 1/1 0 24 33 1080K 340K 1656K 2462 bash 0.0 00:00.00 1 0 17 24 360K 856K 1024K2461 login 0.0 00:00.00 1 0 22 53 468K 312K 1572K2460 mdworker 0.0 00:00.07 3 1 49 64 1532K 14M 3336K2447 _progres 34.0 00:07.41 1 0 22 32 1580K 340K 2156K 2444 bash 0.0 00:00.00 1 0 17 24 364K 856K 1028K2443 login 0.0 00:00.01 1 0 22 53 468K 312K 1572K2407 mdworker 0.0 00:01.41 3 1 50 81 9208K 15M 39M

nmonnmon 3=Top-by-CPU-use Host=usqadnode1 Refresh=8 secs 18:06.26CPU-Utilisation-Small-View EntitledCPU= 3.00 UsedCPU= 0.108Logical CPUs 0----------25-----------50----------75----------100CPU User% Sys% Wait% Idle%| | | | | 0 48.3 49.7 0.0 2.0|UUUUUUUUUUUUUUUUUUUUUUUsssssssssssssssssssssssss>| 1 26.6 38.5 0.0 35.0|UUUUUUUUUUUUUsssssssssssssssssss >| 2 0.0 36.0 0.0 64.0|sssssssssssssssss >| 3 0.0 26.1 0.0 73.9|sssssssssssss > | 4 0.0 24.5 0.0 75.5|ssssssssssss > | 5 0.0 26.8 0.0 73.2|sssssssssssss> | 6 0.0 25.6 0.0 74.4|ssssssssssss > | 7 0.0 35.1 0.0 64.9|sssssssssssssssss > | 8 0.0 23.8 0.0 76.2|sssssssssss > | 9 0.0 24.4 0.0 75.6|ssssssssssss > | 10 0.0 22.9 0.0 77.1|sssssssssss > | 11 0.0 24.9 0.0 75.1|ssssssssssss > | +-----------|------------|-----------|------------+

taskmgr

Windows Task Manager

View -> CPU History -> One Graph Per CPU

Memory Utilization

• How much memory am I really using?• Operating systems “double count” shared

memory…

nmonMemory Physical PageSpace | pages/sec In Out FileSystemCache% Used 97.2% 3.6% | to Paging Space 6.1 4.9 | (numperm) 54.0%% Free 2.8% 96.4% | to File System 3596.3 441.6 | Process 23.0%MB Used 31834.2MB 1024.4MB | Page Scans 256879.7 | System 20.1%MB Free 933.8MB 27135.6MB | Page Cycles 21.4 | Free 2.8%Total(MB) 32768.0MB 28160.0MB | Page Steals 3785.5 | ------ | Page Faults 1785.7 | Total 100.0%------------------------------------------------------------ | numclient 54.0%Min/Maxperm 952MB( 3%) 28545MB( 87%) <--% of RAM | maxclient 87.1%Min/Maxfree 960 1088 Total Virtual 59.5GB | User 74.0%Min/Maxpgahead 2 16 Accessed Virtual 14.4GB 24.2%| Pinned 19.1%

top

top - 10:51:19 up 408 days, 2:18, 1 user, load average: 7.42, 7.17, 7.10Tasks: 696 total, 1 running, 695 sleeping, 0 stopped, 0 zombieCpu(s): 6.9%us, 0.9%sy, 0.0%ni, 85.0%id, 7.2%wa, 0.0%hi, 0.0%si, 0.0%stMem: 4049052k total, 4022004k used, 27048k free, 24352k buffersSwap: 2104472k total, 156k used, 2104316k free, 2055292k cached

free/home/tom> free total used free shared buffers cachedMem: 4049052 4022544 26508 0 25140 2059644-/+ buffers/cache: 1937760 2111292Swap: 2104472 156 2104316

sar -r

• Totally useless

Windows taskmgr

taskmgr – Windows 7

Sysinternals - RamMap

Sysinternals - VMMap

Disk Space

• How much free space do I have?

df, du

/home/tom> df -vkFilesystem 1K-blocks Used Available Use% Mounted on/dev/sdb2 478647540 175136968 279196600 39% /udev 2024524 152 2024372 1% /dev/dev/sdc1 480696480 380900480 75377204 84% /data/dev/sdd1 480696480 233559476 222718208 52% /home/dev/sda3 258033592 180175200 64750960 74% /srv/dev/sda2 2100388 4 2100384 1% /windows/C

/home/protop> du –k76 ./bin80 ./etc96 ./hlp364 ./mon28 ./os/HP-UX24 ./os/AIX28 ./os/SunOS28 ./os/Linux24 ./os/OSF1136 ./os96 ./tmp6736 .

df, du

/home/tom> df -vkFilesystem 1K-blocks Used Available Use% Mounted on/dev/sdb2 478647540 175136968 279196600 39% /udev 2024524 152 2024372 1% /dev/dev/sdc1 480696480 380900480 75377204 84% /data/dev/sdd1 480696480 233559476 222718208 52% /home/dev/sda3 258033592 180175200 64750960 74% /srv/dev/sda2 2100388 4 2100384 1% /windows/C

/home/protop> du –k76 ./bin80 ./etc96 ./hlp364 ./mon28 ./os/HP-UX24 ./os/AIX28 ./os/SunOS28 ./os/Linux24 ./os/OSF1136 ./os96 ./tmp6736 .

Also available for Windows from Sysinternals!

Disk Activity

• Which disks are most active?• Are my disks too active?

taskmgr - XP

nmon

Disk Busy Read Write 0----------25-----------50------------75--------100 Name KB/s KB/s | | | | |hdisk4 8% 29 0 |RRRR > |hdisk6 4% 0 263 |WW > |hdisk7 1% 0 156 |W > |hdisk9 6% 47 67 |RRWW |hdisk10 13% 71 205 |RRWWWWW> |hdisk13 27% 257 47 |RRRRRRRRRRRRWWW > |hdisk17 30% 338 78 |RRRRRRRRRRRRRWWW > |hdisk18 5% 26 12 |RRW > |hdisk21 12% 84 104 |RRRWWWW > |hdisk22 2% 16 0 |R | +-------------------------------------------------+

sar -d

sar -d 60 1

HP-UX pluto B.11.23 U ia64 11/14/10

11:22:40 device %busy avque r+w/s blks/s avwait avserv11:23:40 c3t0d0 3.35 0.62 2 31 0.00 14.52 c24t0d5 0.02 0.50 0 0 0.00 1.68 c24t0d2 0.32 0.50 8 128 0.00 0.40 c24t0d3 0.03 0.50 0 1 0.00 6.62 c24t0d4 0.03 0.50 0 0 0.00 7.66 c24t1d1 0.02 0.50 0 0 0.00 6.20

perfmon

perfmon

Disk Capacity

• What can you expect of your disks?

iometer

iometer

randomrd.c

/home/tom> ./randomrd /data/dbappraise/dbappraise_20.d1 8192 5 500

iterations reads time(sec) reads/sec 1 500 3.9822 125.56 2 1000 3.9961 125.12 3 1500 3.9709 125.92 4 2000 4.0627 123.07 5 2500 3.9150 127.71

Progress Temporary Files

• Who do the files in -T belong to?• What temp files does a process have open?

Progress Temporary Files# ls -l /protemptotal 64-rw-r--r-- 1 tom users 32768 2010-11-13 16:21 DBI1640184576paLIUh-rw-r--r-- 1 tom users 32768 2010-11-13 17:02 DBI2036800256vFfHS7-rw-r--r-- 1 root users 0 2010-11-13 16:21 lbi1Bncn0-rw-r--r-- 1 root users 0 2010-11-13 17:02 lbiXuiCqK-rw-r--r-- 1 tom users 0 2010-11-13 17:02 rcdD5N6Ik-rw-r--r-- 1 tom users 0 2010-11-13 16:21 rcdL2ugyw-rw-r--r-- 1 tom users 0 2010-11-13 16:21 srtfrdJsL-rw-r--r-- 1 tom users 19283638 2010-11-13 17:02 srtzgpRzx

Progress Temporary Files# ls -l /protemptotal 64-rw-r--r-- 1 tom users 32768 2010-11-13 16:21 DBI1640184576paLIUh-rw-r--r-- 1 tom users 32768 2010-11-13 17:02 DBI2036800256vFfHS7-rw-r--r-- 1 root users 0 2010-11-13 16:21 lbi1Bncn0-rw-r--r-- 1 root users 0 2010-11-13 17:02 lbiXuiCqK-rw-r--r-- 1 tom users 0 2010-11-13 17:02 rcdD5N6Ik-rw-r--r-- 1 tom users 0 2010-11-13 16:21 rcdL2ugyw-rw-r--r-- 1 tom users 0 2010-11-13 16:21 srtfrdJsL-rw-r--r-- 1 tom users 19283638 2010-11-13 17:02 srtzgpRzx### fuser -u /protemp/*/protemp/DBI1640184576paLIUh: 15692(tom)/protemp/DBI2036800256vFfHS7: 17980(tom)/protemp/lbi1Bncn0: 15692(tom)/protemp/lbiXuiCqK: 17980(tom)/protemp/rcdD5N6Ik: 18972(tom)/protemp/rcdL2ugyw: 17980(tom)/protemp/srtfrdJsL: 15692(tom)/protemp/srtzgpRzx: 17980(tom)

fuser, lsof# lsof -p 17980COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME_progres 17980 tom cwd DIR 8,49 8192 33652801 /home/tom_progres 17980 tom rtd DIR 8,18 4096 2 /_progres 17980 tom txt REG 8,18 10444522 5867036 /usr/dlc/bin/_progres_progres 17980 tom mem REG 0,0 0 [heap] (stat: No such file or dir)_progres 17980 tom mem REG 8,18 133423 20381698 /lib64/ld-2.4.so_progres 17980 tom mem REG 8,18 217016 52118852 /var/run/nscd/passwd_progres 17980 tom mem REG 8,18 114562 20381728 /lib64/libpthread-2.4.so_progres 17980 tom mem REG 8,18 19808 20381713 /lib64/libdl-2.4.so_progres 17980 tom mem REG 8,18 49642 20381706 /lib64/libcrypt-2.4.so_progres 17980 tom mem REG 8,18 404881 20381716 /lib64/libm-2.4.so_progres 17980 tom mem REG 8,18 970192 5575786 /usr/lib64/libstdc++.so.6.0.8_progres 17980 tom mem REG 8,18 58128 20381823 /lib64/libgcc_s.so.1_progres 17980 tom mem REG 8,18 1505121 20381702 /lib64/libc-2.4.so_progres 17980 tom 0u CHR 136,2 4 /dev/pts/2_progres 17980 tom 1u CHR 136,2 4 /dev/pts/2_progres 17980 tom 2u CHR 136,2 4 /dev/pts/2_progres 17980 tom 3r REG 8,18 1483758 5866996 /usr/dlc/promsgs_progres 17980 tom 4r REG 8,18 943825 5866994 /usr/dlc/convmap.cp_progres 17980 tom 5u REG 8,49 0 33654422 /home/tom/lbiJaKarj (deleted)_progres 17980 tom 6u REG 8,49 19283638 33654423 /home/tom/srtllSsLf (deleted)_progres 17980 tom 7u REG 8,49 0 33654424 /home/tom/rcdNAdL5b (deleted)

sysinternals

Resources

• nmon http://www.ibm.com/developerworks/wikis/display/wikiptype/nmon

• sysinternals http://technet.microsoft.com/en-us/sysinternals/default.aspx

• iometer http://www.iometer.org/

• randomrd.c http://dbappraise.com/

Questions?

Thank-you!

topProcesses: 71 total, 2 running, 1 stuck, 68 sleeping, 336 threads 15:30:49Load Avg: 0.40, 0.23, 0.17 CPU usage: 1.13% user, 1.36% sys, 97.49% idleSharedLibs: 8260K resident, 9820K data, 0B linkedit.MemRegions: 15384 total, 855M resident, 30M private, 679M shared.PhysMem: 918M wired, 1587M active, 587M inactive, 3092M used, 5100M free.VM: 152G vsize, 1042M framework vsize, 197921(0) pageins, 0(0) pageouts.Networks: packets: 2048759/276M in, 2285978/256M out.Disks: 89504/2506M read, 101810/3748M written.

PID COMMAND %CPU TIME #TH RPRVT RSHRD RSIZE VPRVT VSIZE PPID State FAULTS COW2448 PubSubAgent 0.0 00:00.17 4 7588K 14M 9940K 60M 2436M 136 sleeping 2727 1172447 top 4.0 00:00.76 1/1 1572K 244K 2148K 17M 2378M 2444 running 11679+ 552444 bash 0.0 00:00.00 1 364K 244K 1028K 17M 2378M 2443 sleeping 451 1022443 login 0.0 00:00.01 1 468K 244K 1572K 18M 2379M 226 sleeping 617 1252407 mdworker 0.7 00:01.39 3 9208K 14M 39M 92M 2469M 1 running 528165+ 1602392 cupsd 0.0 00:00.01 3 972K 248K 2292K 57M 2418M 1 sleeping 780 872277 AppleSpell 0.0 00:00.09 2 3300K 14M 4960K 47M 2427M 136 sleeping 1559 942273 Safari 0.0 00:15.84 9 52M 65M 74M 230M 4919M 136 sleeping 156817 1581773 ssh-agent 0.0 00:00.12 2 972K 280K 2312K 42M 2411M 136 sleeping 2144 95297 Preview 0.0 00:13.92 2 18M 54M 46M 55M 2781M 136 sleeping 28161 788293- Microsoft AU 0.0 00:00.23 2 948K 1400K 2108K 49M 891M 136 sleeping 2011 143289- Microsoft Ex 0.1 00:31.66 7 37M 95M 58M 88M 1128M 136 sleeping 29079 1554278 Firewall 0.0 00:00.00 1 360K 244K 1304K 10M 2379M 136 sleeping 485 85226 Terminal 1.0 00:17.06 5 20M 39M 35M 65M 2752M 136 sleeping 57884+ 525181- db_daemon 0.0 00:00.10 1 856K 2316K 2848K 18M 589M 1 sleeping 847 113179- gconfd-2 0.0 00:00.14 1 400K 2468K 2236K 18M 589M 1 sleeping 716 128177- VMware Fusio 0.0 00:00.25 3 1168K 18M 3992K 50M 911M 136 sleeping 2902 219

Network Activity

netstat

perfmon, control panel

sysinternals

Kernel Resources

sar

perfmon, sysinternals

Common Use Cases

• CPU Utilization• Memory Utilization• Disk Space• Disk Activity• Disk Capacity

• Progress Temp Files• Runaway Processes• System Calls• Network Activity• Kernel Resources