pcp
Post on 17-Nov-2014
251 Views
Preview:
DESCRIPTION
TRANSCRIPT
Red Hat Enterprise Linux 7Performance Co-Pilot : Overview
Agenda:
➢ Objective of performance monitoring➢ How is this typically done ?➢ Introduction to Performance Co-Pilot➢ Component and Architecture
➢ Useful performance monitoring tools➢ Demo➢ Knowledge sources
Objective of performance monitoring
➢ Early detection of a ( potential ) problem.
➢ Rapidly drill down & pinpoint issue in specific program.
How is this typically done ?
➢ System log files ( rsyslog/syslog-ng/journald )
➢ Native performance monitoring tools ( top/iostat/vmstat/ps )
➢ Mixture of scripting languages ( bash/perl/python )
➢ Specific tools vary per platform
Why we need another Performance Monitoring Tool ?
➢ Ability to monitor performance statistics of a local or remote machines in a complex network.
➢ Ability to record and replay performance statistics.
➢ Ability to monitor multiple hosts performance statistics concurrently.
➢ Customization of clients application to collect specific metrics.
➢ Performance regression detection.
➢ Evaluation of effects of an operating system upgrade etc.
Introduction to Performance Co-Pilot
➢ PCP is a system level performance monitoring and performance management toolkit.
➢ PCP provides a range of services that may be used for collection, monitoring and analysis of system metrics.
➢ The Performance Co-Pilot includes many facilities for creating and replaying archive logs that capture performance information.
➢ Cross platform support : Linux, Mac OS and Windows.
➢ End-to-end : Hardware, Core OS, services and applications.
Introduction: Performance Co-Pilot
➢ Distributed architecture
Monitoring of local and remote nodes.
➢ Real-time or retrospective
Live system or archive.
➢ Pluggable
New agents system metrics within PCP.
Performance Co-Pilot - Components
Collectors
● Collect and export performance metrics● Performance Metric Domain Agents (PMDA)● Performance Metric Collection Daemon (PMCD)
Consumers
● Record, visualise, monitor and analyse performance data● Consume data either in realtime or replay historical data from
archive logs
Note: Hosts may operate as collectors, consumers or both
● Multiple consumers may connect with one or more collectors
Performance Co-Pilot - Collectors
Performance Metric Domain Agents (PMDA)
● Extracts & exports metric data from a system component● Communicates with pmcd on local system
Performance Metric Collection Daemon (PMCD)
● One pmcd process per host● Coordinates handling of fetch requests between consumer
applications and agents● Listens for connections from localhost & remote clients
● Authenticated & encrypted connection options
Performance Co-Pilot : Consumers● pmlogger
● Utility to capture and store metrics exported by PMCD● Concurrent logging of data from local and remote hosts● Archive playback by other consumer tools (eg pmchart, pmval)● Tools for log archival, log rotation etc● Operates on data in realtime only
● pminfo - display PCP metrics available on a host or in an archive
● pmchart - GUI utility providing graphical display of PCP data
● pmstat - vmstat-like utility
● pmatop - top-like utility
● pmie - inference engine and alerting utility
● pmval - display a metric
Performance Co-Pilot - Architecture
Webserver
DBMS
Network
Kernel
PMDAs
PMCD
pmchart
pmstat
pmlogger
Consumers
Archive ClientsConsumers
Performance Co-Pilot – Distributed Architecture
AgentsPMCD
PMDA'sPMCD
Host 2
PMDA'sPMCD
Host 3
PMDA'sPMCD
Host 1
ClientsConsumers
Clients
Host 4
Consumers
Installation:
➢ To install pcp and pcp-gui package, type the following command:
# yum install pcp pcpgui
“pcp” package provides a framework and services to support system-level performance monitoring and performance management.
“pcp-gui” package provides a visualization tools for the Performance Co-Pilot toolkit.
Services:To start pmlogger and pcp services, type the following command:
# chkconfig pmlogger on
# service pmlogger start
Starting pmlogger ...
# service pcp start
Starting pmcd ...
Starting pmlogger ...
To check the status of pcp service, type the following command:
# service pcp status
Checking for pmcd: running
Checking for pmlogger: running
Note: Bug 1044682 - pcp should use systemd
Verification:pcp - Summary of PCP installation
# pcp
Performance CoPilot configuration on localhost.localdomain:
platform: Linux localhost.localdomain 3.10.0121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64
hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM
timezone: IST5:30
pmcd: Version 3.8.101, 6 agents
pmda: pmcd proc xfs linux mmv jbd2
# pcp h `hostname`
Performance CoPilot configuration on localhost.localdomain:
platform: Linux localhost.localdomain 3.10.0121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64
hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM
timezone: IST5:30
pmcd: Version 3.8.101, 6 agents
pmda: pmcd proc xfs linux mmv jbd2
Configuration files:
Main configuration file for pcp:
/etc/pcp.conf
Main configuration file for pmcd:
/etc/pcp/pmcd/pmcd.conf
Default pmlogger config file:
/etc/pcp/pmlogger/config.default
PCP archive logging configuration/control:
/etc/pcp/pmlogger/control
Note: pmlogger is utility to capture and store metrics exported by PMCD
Log files:
Log file directories for pcp components:
/var/log/pcp/pmcd
/var/log/pcp/pmie
/var/log/pcp/pmlogger
/var/log/pcp/pmmgr
/var/log/pcp/pmproxy
/var/log/pcp/pmwebd
Directory Organization for Archive Log Files
Performance monitoring tools:
➢ For all PCP monitoring tools, metrics values may come from a real-time feed (i.e. from pmcd on some host), or from an archive log.
➢ Performance monitoring tools available in Performance Co-Pilot (PCP).
pmstat
pmatop
pmcollectl
pmval
pmchart
pminfo
Performance monitoring tools:
pmstat - vmstat-like utility, intended to monitor system performance at the highest level.
# pmstat
# pmstat t 1 T 3
# pmstat a /var/log/pcp/pmlogger/localhost.localdomain/20140607.20.20.0
Performance monitoring tools:
pmatop - top-like utility
#pmatop
Record Mode:pmatop w rawfile [ interval [ samples ]]
Playback Mode:pmatop r [ rawfile ] [g|m] [L linelen] [h host]
Eg:# pmatop w atop.out 1 4# pmatop r atop.out
Performance monitoring tools:
pmcollectl - System-level performance monitoring utility that records or displays specific operating system data for one or more sets of subsystems.
Basic Systemlevel performance monitoring:# pmcollectl
Subsystem Specific performance monitoring:# pmcollectl sm# pmcollectl sm –verbose# pmcollectl smcdn –verbose
Record Mode:# pmcollectl f perfdata c 3
Playback Mode: # pmcollectl p perfdata
Performance monitoring tools:
pminfo - Display PCP metrics available on a host or in an archive.
Full list of all available metrics: # pminfo F
Fetch and print values for all or specific metrics:# pminfo f# pminfo f <metric># pminfo F a /var/log/pcp/pmlogger/localhost.localdomain/20140607.20.56.0
Eg:# pminfo f proc.nprocs disk.dev.read filesys.free
Descriptions of all or specific metrics:# pminfo T d # pminfo T d <metric>
Eg:# pminfo T d mem.util.cached# pminfo T d proc.nprocs disk.dev.read filesys.free
Performance monitoring tools:
pmval - Dumps the current values for the named performance metrics. # pmval <metric>
Eg:# pmval proc.nprocs# pmval kernel.all.load# pmval t 2sec s 4 kernel.percpu.cpu.idle
Performance monitoring tools:
pmchart - GUI utility providing graphical display of PCP data
Knowledge sources:
Performance Co-Pilot (PCP)
http://oss.sgi.com/projects/pcp/
PCP Manual:
http://www.performancecopilot.org/pcp.git/man/html/index.html
Blog:
Exploratory Performance Analysis with Performance Co-Pilot
http://developerblog.redhat.com/2013/11/19/exploratory-performance-pcp/
Knowledge sources:
Videos:
Using Performance Co-Pilot to monitor SNMP devices - Hamish Coleman
https://www.youtube.com/watch?v=2azBcj8QUdI
Exploratory Performance Analysis with Performance Co-Pilot
https://www.youtube.com/watch?v=zrAjevr8_Ds
Regression Analysis with Performance Co-Pilot
https://www.youtube.com/watch?v=Z9fSymDfuvQ
Questions ?
top related