practical psft ppmukoug2008 © practical guidance on the use of peoplesoft performance monitor david...
TRANSCRIPT
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 1
Practical Guidance on the Use of PeopleSoft Performance Monitor
David KurtzGo-Faster Consultancy Ltd.
www.go-faster.co.uk
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 2
• Oracle Database Specialist– Independent consultant
• System Performance tuning– PeopleSoft ERP
– Oracle RDBMS
• Book– www.psftdba.com
Who Am I?
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 3
Agenda
• Instrumentation – Oracle RDBMS– PeopleSoft PeopleTools
• Using PeopleSoft Performance Monitor– Problems– Performance Trace– Digging in the data
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 4
Resources
• If you can’t hear me say so now.• Please feel free to ask questions as we go
along.• The presentation will be available from
• Conference website• www.go-faster.co.uk
• Article in UKOUG Oracle Scene magazine
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 5
Performance Tuning
• We do not use ratio based tuning any more.– Especially the buffer cache hit ratio
• We do use timed event based tuning.
• Further reading:– YAPP – Kolk, Yamaguchi, and Viscusi– Optimising Oracle Performance - Millsap & Holt
• www.method-r.com, www.hotsos.com, www.oreilley.com
– The Goal – Eli Goldratt
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 6
Performance Instrumentation
• Oracle does understand instrumentation.– Instrumentation is built in throughout the
database kernel.
• PeopleTools 8.44 upwards includes Performance Monitor.– Instrumentation built into PeopleSoft
technology.
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 7
Oracle RDBMS Instrumentation
• Dynamic Performance Views– Statspack – AWR / ADDM / ASH (licensed extras)
• Trace/Dump to operating system files– Enhanced by setting events – Profilers
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 8
Event 10046 Level 8
• a.k.a. SQL*Trace with timed event information• Includes
– Every SQL Statement• How long it took to parse/execute/fetch
– Row source operation list• How long each operation took
– STATISTICS_LEVEL = ALL
– Every event for which the database waits
• Can then profile with TKPROF or other profilers
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 9
What does this instrumentation do for us?
• If you have a performance problem, then you can determine exactly what the database is doing, and how long that is taking.– It can also prove that the problem is not located
in the database!
• Hence, you can work out what to do about it.
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 11
You probably don’t need event 10046!
• SQL*Net Message from Client– Idle Event?– Database is Idle, but is the user idle?– Or is something else active in the technology
stack (so the user is still waiting)?
• This event cannot distinguish between these conditions.
You probably won’t get all you need from event 10046!
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 12
Timed Event Information
• Oracle timed events tell us about the database.
• We need similar information for every element in the technology chain.
• PeopleSoft realised this and instrumented their entire technology stack.– And they got it right!
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 13
PeopleTools Performance Utilities
• Instrumentation in PeopleTools 8.4– Query Statistics
• 8.44, usable from 8.45
– PeopleSoft Ping• 8.42, back-ported to 8.19
– Performance Monitor• 8.44
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 14
Performance Monitor
• PeopleTools 8.44 – Fully instrumented– Including a timed-event interface– Event 10046 for the application– Useful PeopleBook– Additional analytics in PT8.45
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 15
Performance Monitor Architecture
• Based upon existing PeopleSoft technology• Monitored System
– Send information to servlet in monitoring system
• Monitoring System– Monitor servlet writes results to database via
PSPPMSRV process in application server • Ideally PeopleTools only system database
• This minimises measurement intrusion effect
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 16
WebServer
WebServer
Application Server
(application logic)
Application Server
(application logic)
APPQ PSAPPSRVDBMS
(application data & meta-
data
DBMS
(application data & meta-
data
SQLTuxedo
Messagehttp / https
Browser
(presentation & JavaScript)
Browser
(presentation & JavaScript)
Monitoring System
PIAServlet
PSPPMSRV
Monitored System
Web Server
(presentation logic)
Web Server
(presentation logic)
PIAServlet
DBMS
(application data & meta-
data
DBMS
(application data & meta-
data
Tuxedo Message
http / https
Browser
(presentation & JavaScript)
Browser
(presentation & JavaScript)
ScreenPaint
JavaScript
Application Server
(application logic)
Application Server
(application logic)
APPQ PSAPPSRV
PSMONITORSRV
SQL
PPMIServlet
MonitorServlet
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 17
Performance Monitor Architecture
• Instrumentation in – Application Server processes
• PSMONITORSRV collects host resource statistics– Memory
– CPU
– Process Scheduler– PIA servlet
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 18
Performance Monitor Metrics
• Transactions– User activities in PIA that cause communications with
application server
– Sampled
– Enabled to form a trace
• Events– Periodic samples
– Usually initiated by monitoring agents
– eg. CPU, Tuxedo counters
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 19
Performance Monitor Transactions
• User activity in PIA• Performance
Monitoring Unit– Hierarchy of
transactions
• Similar to Oracle event 10046 trace– recursive actions
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 20
Transactions
• Stored to PSPMTRANSCURR table– As PMUs are closed moved to
PSPMTRANSHIST– Later deleted or archived to
PSPMTRANSARCH
• ERD downloadable from Customer Connection
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 21
ERD of TransactionPSTRANSHIST
(C)
PSPMSYSDEFN(B)
PSPMAGENT(A)
PSPMCONTEXTDEFN(C1)
PSPMMETRICDEFN(M1)
PSPMCONTEXTDEFN(C2)
PSPMCONTEXTDEFN(C3)
PSPMMETRICDEFN(M2)
PSPMMETRICDEFN(M3)
PSPMMETRICDEFN(M4)
PSPMMETRICDEFN(M5)
PSPMMETRICDEFN(M6)
PSPMMETRICDEFN(M7)
PM_CONTEXTID_n
(n=1-3)
PM_SYSTEMID
PM_TRANS_DEFN_SET, PM_TRANS_DEFN_ID
PM_AGENTID
PM_METRICIDn
(n=1-7)
search criteria
PSPMTRANSDEFN(E)
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 22
Metrics
• Metric IDs specified on transaction definition PSPMTRANSDEFN– Metrics Types defined on PSPMMETRICDEFN
• Type 1: Counters (including timers)– Metric 4: Total Servlet Request time (ms)
• Type 2: Gauges– Metric 102: %CPU Used
• Type 3: Numeric Identifier– Metric 20: HTTP response code
• Type 4: String Identifier– Metric 27: File Name
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 23
Transaction 101
• Reported at entry and exit of PIA servlet – Context 1
Action=View Page
– Context 2IP Address=10.0.0.3
– Context 3Session ID=AN7tpzSwpZc4kt9k8 . . .
– Additional Descriptionhttp://go-faster-3:7201/psc/ps/EMPLOYEE/HRMS/c/UTILITIES.PTPERF_TEST.GBL
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 24
Transaction 101
• 4 metrics– Metric 19: Response Size (bytes)
=17613
– Metric 20: Response Code =200
– Metric 22: Static Content Count =0
– Metric 23: Is this a Pagelet? =0
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 25
Transaction Query ResultsPM_TOP_INST_ID PM_INSTANCE_ID PM_PARENT_INST_ID DBNAMEPM_HOST_PORTPM_DOMAIN_NAME PM_AGENT_TYPEPM_INSTANCE PM_AGENT_STRT_DTTM PM_MON_STRT_DTTMOPRID PM_PERF_TRACE PM_PROCESS_IDPM_TRANS_DEFN_ID DESCR60'CONTEXT1:'||C.PM_CONTEXTID_1||'-'||C1.PM_CONTEXT_LABEL||'='||C.PM_CONTEXT_VALUE …PM_TRANS_DURATION'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 …PM_ADDTNL_DESCR-------------------------------------------------------------------------------- 824633721163 824633721163 0 HR88go-faster-3:7201:7202ps WEBSERVER-1 16:12:07 14.06.2004 16:12:09 14.06.2004PS PS: 2004-06-14 16:01:11 0 101 Reported at entry and exit of PIA servletContext1:3-Session ID=AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt!1963244185!1087224685145Context2:2-IP Address=10.0.0.3Context3:1-Action=View Page 1322Metric1:Response Size (bytes)=17613Metric2:Response Code=200Metric3:Static Content Count=0Metric4:Is this a Pagelet?=0Metric5:=0Metric6:=0Metric7:=http://go-faster-3:7201/psc/ps/EMPLOYEE/HRMS/c/UTILITIES.PTPERF_TEST.GBL
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 26
Events
• Certain events are defined for which the monitor agents collect metrics– Regular cycle
• Host resources
– In response to user action• PeopleSoft Ping
– On an exception• Jolt Exception or Query Timeout
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 27
Events
• Do not have an explicit context– Collecting agent provide context
• Stored in PSPMEVENTHIST– Later deleted or archived to PSPMEVETARCH
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 28
ERD of EventsPSEVENTHIST
(C)PSPMEVENTDEFN
(E)
PSPMSYSDEFN(B)
PSPMAGENT(A)
PSPMMETRICDEFN(M1)
PSPMMETRICDEFN(M2)
PSPMMETRICDEFN(M3)
PSPMMETRICDEFN(M4)
PSPMMETRICDEFN(M5)
PSPMMETRICDEFN(M6)
PSPMMETRICDEFN(M7)
PM_AGENTIDPM_EVENT_DEFN_SET, PM_EVENT_DEFN_ID
PM_SYSTEMIDsearch criteria
PM_METRICIDn
(n=1-7)
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 29
Event Query ResultsDBNAME PM_HOST_PORTPM_AGENT_TYPE PM_DOMAIN_NAMEPM_INSTANCE PM_AGENT_DTTM PM_INSTANCE_IDPM_EVENT_DEFN_ID DESCR60'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1'METRIC2:'||M2.PM_METRICLABEL||'='||C.PM_METRIC_VALUE2'METRIC3:'||M3.PM_METRICLABEL||'='||C.PM_METRIC_VALUE3'METRIC4:'||M4.PM_METRICLABEL||'='||C.PM_METRIC_VALUE4'METRIC5:'||M5.PM_METRICLABEL||'='||C.PM_METRIC_VALUE5'METRIC6:'||M6.PM_METRICLABEL||'='||C.PM_METRIC_VALUE6'METRIC7:'||M7.PM_METRICLABEL||'='||C.PM_METRIC_VALUE7PM_ADDTNL_DESCR--------------------------------------------------------------------------------HR88 go-faster-3:7201:7202WEBSERVER ps-1 16:12:08 14.06.2004 824633721166 600 PSPING metrics fowarded from browserMetric1:Network Latency (ms)=435Metric2:WebServer Latency (ms)=100Metric3:AppServer Latency (ms)=561Metric4:DB Latency (millisecs)=451Metric5:=0Metric6:=0Metric7:IP Address=10.0.0.3PS;AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt!1963244185!1087224685145
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 30
Using PeopleSoft Performance Monitor
• Bugs & Fixes
• Performance Tuning the Performance Monitor
• Purge Process
• Performance Trace Demo
• Network Latency
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 31
Bugs and Fixes
• Ports left in CLOSE_WAIT on Monitored System– Until Unix/Linux runs out of ports
• Windows doesn’t have a limit on number of ports
– ‘Application Server is Down’ message
• Affects PT8.49– From at least 8.49.06– Fixed in 8.49.15
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 32
Tuning Performance Monitor
• Some of the delivered analytics do not perform well with even moderate data volumes– Set up the monitoring system to self monitor– Then you can generate PPM traces on the
analytics– You will need additional indexes
• http://blog.psftdba.com/2006/04/performance-tuning-performance-monitor.html (YMMV)
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 33
Purge Process
• Data normally held in history tables– PSPMTRANSHIST, PSPMEVENTHIST
• Clone tables– PSPMTRANSHISTCL, PSPMEVENTHISTCL
• PPM writes to tables specified in PSPMTABLEMAP– Archive process switches this to clone tablesselect * from pspmtablemap;PM_TRANS_TBL_NAME PM_EVENT_TBL_NAME------------------ ------------------PSPMTRANSHIST PSPMEVENTHIST
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 34
PSPMTABLEMAP
• Archive/Purge switches PPM destination– Prevents concurrent INSERT and DELETE/Query
operations
– Saves read consistency problems on Oracle
– Saves page locks on other databases
• PPM appears not to collect data during this process– But it is written to clone tables
– Archive process moves it to main hist tables after purge
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 35
PSPMARCH Job Duration
0:00
0:30
1:00
1:30
2:00
Sun 1.6.08 Sun 8.6.08 Sun 15.6.08 Sun 22.6.08 Sun 29.6.08 Sun 6.7.08 Sun 13.7.08
Start Date
Du
rati
on
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 36
Purge Process Can’t Keep Up
• Platform generic query leads to full scan – Even if data has been deleted (manually) high
water marks (HWM) on tables not reset
• Customisation– Oracle specific statement– May need to rebuild HIST tables to reset HWM
• In which case manually set PSPMTABLEMAP to clones, rebuild history tables, run archive process
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 37
Performance Fix for Purge
• Vanilla Code&TransHistSQL.Open("SELECT …AND %DateTimeDiff(X.PM_MON_STRT_DTTM, %CurrentDateTimeIn)
>= (PM_MAX_HIST_AGE * 24 * 60)");
• Expands to … AND ROUND((CAST(( SYSDATE) AS DATE) -
CAST((X.PM_MON_STRT_DTTM) AS DATE)) * 1440, 0) >= (PM_MAX_HIST_AGE * 24 * 60)
• My Suggestions… AND X.PM_MON_STRT_DTTM < SYSDATE - Z.PM_MAX_HIST_AGE
• See blog entry http://blog.psftdba.com/2008/05/performance-tuning-performance-monitor.html
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 38
How much data?
• Control sampling– Proportion of transactions collected
• Depends upon activity on system • On busy self-service system as little as 1in 5000
– Event sampling frequency• For each agent• 5 minutes – 15 minutes• Depends on whether you want to be able to see
shortlived behaviours.
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 39
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 40
Agent Filter Levels
• Controls amount of detail recorded in PMUs– 4- standard– 5 – verbose
• Includes SQL fetch operations
• Performance penalty
– 6 - debug
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 41
Agent Filter Levels
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 42
Analytics: System Performance
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 43
Analytics: Top Components
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 44
Performance Trace
• Generates a group of PMUs for activity in a user session– Choose an ID to
identify records later
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 45
Performance Trace
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 46
Performance Monitoring Unit
• Look at PMU Tree
• Demonstration
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 47
Network Latency
• Most transactions are sampled
• But three transactions are always recorded• See PMTRANSDEFN.PM_SAMPLING_ENABLE
– 108: User Session logout, expiration, timeout, or error
– 109: User Session began (user logged in)– 116: Redirected round trip time (network
latency)
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 48
Transaction 116
• Network round trip from webserver to browser and back again– Includes network transmission time– Browser response time– Client IP address
• Although that could be load balancer or NAT
– Operator ID• LOCATION from HR database?
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 49
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 50
Analysis by Client IP
• IP addresses are of Routers not Clients
Mon Jun 30 page 1 Login Durations by Client IP
IP Address MIN AVG MED MAX VAR NUM_EVENTS---------------- ------ ------ ------ ---------- ------------ ----------147.149.127.223 0.000 0.128 0.094 0.390 18.3 12147.149.1.200 0.047 0.356 0.321 3.046 57.8 290147.149.126.149 0.015 0.604 0.156 14.077 1,835.0 1288192.168.1.171 0.032 0.692 0.157 6.392 2,064.6 1226147.149.126.147 0.031 0.712 0.156 15.062 2,322.9 1302192.168.1.172 0.031 0.742 0.172 61.321 5,218.6 1234192.168.1.170 0.047 0.748 0.172 12.989 2,342.9 1274147.149.126.148 0.031 0.762 0.141 44.918 3,963.8 1300193.113.139.184 0.422 5.380 5.423 20.951 589.8 803
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 51
HR location of OPRID
• First three lines are different network topology• Operator may not actually be at stated location
– esp home workers
Mon Jun 30 page 1 Login Durations by Location (min 10 logins)
LOCATION CITY Cty MIN AVG MED MAX VAR EVENTS---------- ---------------- --- ------ ------ ------ ------- ---------- ------EXT-00-IN Bangalore IND 0.109 5.488 5.468 61.321 5,099.0 688EXT-00-CZ Prague CZE 0.109 4.991 5.218 20.951 2,125.1 555EXT-MK-UK Milton Keynes GBR 0.062 0.862 0.281 6.578 3,100.8 224
HOME Home Worker* GBR 0.031 0.299 0.188 9.109 377.7 489
INT-L3 Liverpool GBR 0.062 0.211 0.110 2.406 140.6 40INT-MK Milton Keynes GBR 0.032 0.233 0.078 1.266 114.4 30INT-WS Walsall GBR 0.031 0.086 0.063 0.359 3.5 46
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 52
Login Redirect Duration
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Fri 27.6.08 00:00 Sat 28.6.08 00:00 Sun 29.6.08 00:00 Mon 30.6.08 00:00 Tue 1.7.08 00:00 Wed 2.7.08 00:00 Thu 3.7.08 00:00 Fri 4.7.08 00:00
Time
Du
rati
on
(s)
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 53
Summary
• Very impressive feature rich piece of code– It could also put me out of business!
• In house support tool– An end user could be enabled to collect a
performance trace.
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 54
My Questions
• Will Fusion have a Performance Monitor?– I think it should
• Will it be separately licensed product?– PeopleSoft’s performance monitor isn’t
Your Questions?
Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk 56
Practical Guidance on the Use of PeopleSoft Performance Monitor
David KurtzGo-Faster Consultancy Ltd.
www.go-faster.co.uk