www.clearpointmetrics.com metrics 101 may, 2006. © 2005-2006 clearpoint metrics proprietary &...
TRANSCRIPT
www.clearpointmetrics.com
Metrics 101
May, 2006
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 2© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
Outline
What is a metric ? What makes a metric good ? What makes metrics so hard ? How are metrics used ? How are metrics mapped to the business ? What are some examples of good metrics ? What are some easy, automatable metrics ? What are some good ways to organize metrics ? Where do metrics get their data ? What is a well-managed metric ? What are key components of a well-managed metric ? What is the lifecycle of a metric ? What does a purpose-built tool for designing metrics look like ? What is a default view of metric ? What can a custom scorecard on a specific topic look like ?
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 3© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What is a Metric ?
Measurement – Generated by counting Measurements provide quantitative observations of
discrete factors, in isolation
Metric – Derived through analysis applied to measurements Provide quantitative data about a target process or
asset in order to achieve an explicit purpose Truly useful metrics provide the insight needed to
make better decisions.
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 4© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What Makes a Metric Good ?
Flexibility Measurement technique Sources employed Publication mechanism
Accuracy Repeatable Auditable
Context Correlates measurements across multiple sources Comparability across multiple dimensions (e.g. time, location) Relevance to the Business (e.g. organization, applications, LoB’s) Assignable to someone (e.g. accountability)
Transparency Assumptions well documented Data sources explicitly identified Models, analytics, algorithms completely and unambiguously defined
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 5© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What Makes Metrics so Hard ?
RouterCisco3COMLucentNortelJuniper
FirewallCheckPointJuniperCiscoSymantec
AntivirusSymantecMcAfeeTrend MicroPandaSophos
Web CacheInktomiPersistenceF5, Cisco
Web ServerApacheIISNetscapeiPlanet
NetworkSystemMgtHP OpenViewIBM TivoliCA UnicenterCiscoRemedy
App ServerBEAOracleWebspere
ApplicationMiddlewareBEAWebsphereTibco
ERPSAPPeopleSoftOracleUser-written
CRMSeibelPeopleSoftOracle
DatabasesOracleSQL ServerDB2
DMZ Middle Tier Back-End
Partners & Suppliers
Heterogeneous and dispersed silo’s of vital IT information Never the same for any two organizations Difficult to fuse together silo’s and map results to a business context Challenging to express Exacting to communicate effectively
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 6© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
How are Metrics Used ?
To answer strategic questions: Is security getting better or worse? What is the value of a specific security investment? Where is the point of diminishing returns on a security
investment? What options exist and what are their consequences in terms of
security or operational risk? Are risk avoidance policies being followed?
To justify allocation of resources To drive positive change, awareness, accountability To provide hard quantitative evidence of the existence,
execution, coverage, effectiveness of controls To make better decisions
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 7© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
How are Metrics Mapped to the Business ?
80-20 Rule: Generic versus Specific Some Techniques:
Fusion across islands of data Comparative analysis and decomposition of values across
business-relevant categories such as Line-of-Business, office location, or asset class
Thresholds and goal-attainment models Coverage, efficiency, and effectiveness models Correlation between related metrics Complexity and variability models Weighting models for risk
Often done manually, but often (at least partially) amenable to automation.
Automation expands scope & capacity while improving accuracy, regularity, accountability, and repeatability
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 8© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What are Some Examples of Good Metrics ?
1. Latency between employee termination and de-provisioning of all access to applications Context: IAM Efficiency Use: SSO Project Evaluation KPIs: Accounts/User, Support Time/Account, Support Cost/User –
min, max, mean, variance
2. Vulnerability Scan Coverage for past 12 months Context: Vulnerability Management Effectiveness Use: Comparison of two scanners KPIs: PercentScanned/Scan/Scanner – min, max, mean, variance
3. False Positive Rate for Event Manager for past 365 days Context: Event Management Accuracy Use: Cost analysis of Incident Response Center workload KPIs: Percent False Positives/Day (signal/noise), Cost/Event
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 9© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What are Some Easy, Automatable Metrics ?
Incidents1. # of incidents and severity2. Mean time to resolve incidents…Hosts7. % hosts adhering to policy8. # of workstations using server
ports10.% of hosts that are portable (i.e.
laptops)…Users21.# of power users by division and
trending23.# adhering to password aging policy25.% with VPN access by user type…Perimeter35.# of internet facing hosts36.# of open ports and intended
application37.# of unused firewall rules…
IT Systems Configuration Turbulence78.# new hosts per time period79.# additional open ports/services per time
period80.# additional users per time period…
Patch Management55.# of vulnerabilities identified by system type56.Latency from patch release to patch
application57.Number of patches applied per time period…Storage & Backup66.% of hosts with managed backups67.hours of backup gap by system type and purpose71.# of restore requests (type/who/critical?)…
Virus Management89.% systems with AV systems90.AV signature age by all dimensions95.# inbound/outbound viruses at perimeter …
Vulnerability Management97. Average time period between scans99. % Machines scanned100.# Vuln’s identified by by type / patch level..
Threats35.# attacks by severity36.# false positives by severity37.# incidents by severity…
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 10© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What are Some Ways to Organize Metrics ?
ThreatsCovers active monitoring and defenses against attacks, such as anti-virus and IDS systemsIncludes metrics around topics such as:
Security Event Management Host Anti-virus systems Network Anti-virus systems IDS Systems Incident Response
Compliance & RiskCovers compliance dimensions across other metrics, such as compliance of financial systems for SOX.Includes metrics around topics such as:
Coverage of Network Management Coverage of automated scanning Coverage of IDS and Anti-virus Risk weighting of systems
VulnerabilitiesCovers vulnerabilities inherent in hosts, such as discovery and remediation of known exploitable vulnerabilities and the application of patches.Includes metrics around topics such as:
Vulnerability Identification and Scanning
Vulnerability Remediation Patch Identification Patch Application
Identify and Access ManagementCovers user access and authentication to the organization’s systems.Includes metrics around topics such as:
Password age and strength User roles and permissions External Access
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 11© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
Where Do Metrics Get Their Data ?
Generic CSV Files Spreadsheets JDBC/ODBC LDAP
Threat Managers ArcSight eSecurity Symantec DeepSight
Vulnerability Managers Qualys QualysGuard ISS Site Protector Tenable/Nessus
Anti-Virus, Anti-Spam Trend Micro McAfee ePO Symantec AV
Network and System Managers Tivoli MSFT MOM HP OpenView
Identity and Access Management MSFT IAM Series Tivoli TIM, TAM, FIM CA/Netegrity Site Minder Symantec L0pht Crack John the Ripper
Incident Management Systems Remedy ARS Peregrine Service Center JourneyX Timekeeper
Human Resource Managers PeopleSoft SAP
Asset Managers MSFT SMS Tivoli Management Framework CA Unicenter Asset Manager LANDesk Management Suite
Application/Storage Security Mgrs Fortify Vontu
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 12© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
Metrics can be Data Producers, too
RiskMgmt.
AssetMgmt. Financial
Mgmt.
SecurityOps
Regulatory& Compliance
HR CRM
Vuln & Threat Info
Sources
NSM PolicyRespCenter &
Help Desk
Regulations
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 13© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What is a Well-Managed Metric ?
A Well-Managed Metric has … Architecture
Components Interfaces to data providers and metrics consumers
Dynamic Concept of Operation Flow of work: Sequencing, Precedence, Schedules Contingency handling
Life Cycle & Operations Management Crisply defined stages Auditable transitions between stages Administration: Fault, Configuration, Accounting/Auditing,
Performance, & Security
Portability with respect to-- Data providers, data consumers Publication method
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 14© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What are Key Components of a Well-Managed Metric ?
General Information Author Description Version Annotations: taxonomies, keywords
Data Sources Reference definition Interface drivers Mapping from at least two sources to reference definition
Business Logic Workflow orchestration Models, Analytics, Calculations
Persistence Logic Schedule Publication Logic
Schedule Notification criteria (Default) Scorecard visualization
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 15© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
MxStudio
Deploy
Metric & Scorecard
Description Package
MXServer
Metrics Result DB
MxPublisher
Publish
Create Calculate Communicate
What is the Lifecycle of a Well-Managed Metric ?
How often will this metric be computed?
What sources will provide the data?
What is the workflow to collect, compute and store results?
Where will results be accumulated over time?
Deliverable: Results
What question will this metric answer?
What data does it need? What calculations and models
will it use? What results will it produce? How should results be
visualized and published?Deliverable: Atomic Package
How often will scorecard editions be published?
What do they look like? Who can see what? Where will scorecards be
delivered? What is the workflow for
annotating & approving scorecards before pub?
Deliverable: Scorecards
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 16© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What Does a Metric Authoring Tool Look Like ?
Hierarchically Organized Metric Catalog
Metric Editors:• General Information• Workflow (shown)• Schedule• Tester/Debugger
Analysis Wizards• Aggregation• Filtration• Transformation• Built-in Functions• Escape to Javascript
Hierarchically Organized Catalogs for Data Sources and Canned Analytics
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 17© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What is a Default Visualization of Metric Results
Hierarchically Organized Scorecard Catalog
Hierarchically Organized Metrics Results
Default Visualization of Metric Results in Chart or Tablular Format
Display Controls, e.g. chart type and time period
© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 18© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential
What can a Custom Scorecard Look Like ?