1 it security metrics a practical approach to measuring information security: measuring security at...
TRANSCRIPT
1
IT Security Metrics
A Practical Approach to Measuring Information Security:
Measuring Security at the System Level
3
IT Security Metrics Training
Audience: Federal IT security personnel with GISRA reporting responsibilities
Goal: To train Federal IT security personnel how to develop metrics that they can use immediately to assist with GISRA reporting
Duration: 3 hours
4
Objectives
After completing this workshop, you will be able to:
• Identify why metrics are important for IT security
• Identify the relationship among GISRA, NIST SP 800-26, and IT Security Metrics
• Describe IT Security Metrics
• Describe metrics development process
• Apply metrics development process by completing a Metrics Form for one of the OMB GISRA reporting requirements for FY02
• Identify metrics-related Roles and Responsibilities
• Describe how to implement a Metrics Program
6
In this section, you will:
• Learn the definition and characteristics of IT Security Metrics
• Identify the difference between Performance Goals and IT Security Metrics
• Learn the seven-step IT Security Metrics Development Process
• Discover the types of information and insights that can be gained from IT Security Metrics
• Complete three examples of IT Security Metrics
7
What are IT Security Metrics?
IT Security Metrics are tools that facilitate decision making and accountability through collection, analysis, and reporting of relevant performance data.
• Based on IT security performance goals and objectives
• Quantifiable
• Obtainable/feasible to measure
• Repeatable
• Provide relevant performance trends over time
• Useful in tracking performance and directing resources
8
Why Measure IT Security?
• Measure successes and failures ofpast and current security investments
• Justify future investments
• Improve accountability to stakeholders
• Ensure appropriate level of mission support
• Determine IT security program effectiveness
• Improve customer confidence
• Government Information Security Reform Act (GISRA)
• Clinger-Cohen Act
• Government Paperwork Reduction Act (GPRA)
• Enable investment targeting to identified areas in need
• Ensure best value from security
• Build confidence in leadership
• Demonstrate improvement to stakeholders
• Play key role in initiating improvementactions based on performance trends
• Enable relevant, realistic, appropriatesecurity procedure modification
• Satisfy regulatory requirements
Financial
Organizational
Regulatory
Requirements BenefitsCategories
9
IT Security Metrics should support IT security goals and objectives
IT Security Performance Goals identify desired results of system security program implementation.
IT Security Performance Objectives enable accomplishment of goals by:
• Identifying strategic practices, defined by security policies, procedures, and controls
• Directing consistent implementation of policies and procedures across the organization
IT Security Metrics monitor accomplishment of goals and objectives by:
• Quantifying the level of implementation of security control objectives and techniques for a system and the effectiveness and efficiency of the controls within the
organization
• Using analysis of collected IT Security Metrics to determine adequacy of security activities and make appropriate business decisions
10
Exercise: Performance Goal or IT Security Metric?
.
.
Program Officials understand the risk to systems undertheir control and determine the acceptable level of risk.
Percentage of system security plans that areupdated annually.
Duties are separated to ensure least privilegeand individuals accountability.
Percentage of systems with automated virus updating.
Data integrity and validation controls are used to provide assurance that the information has not beenaltered and the system functions as intended.
IT Security MetricStatement Performance Goal
13
Stakeholders and Interests
• Anyone within an organization is an IT security stakeholder, though some functions have a greater stake than others:
– CIO– Program Manager/System Owner– Security Program Manager– Resource Manager– Training/Human Resources Personnel
• Each stakeholder needs a set of metrics that provides a view of the organization’s IT security performance within their needs, for a total of no more than 10-20 metrics per stakeholder
• Many IT Security Metrics can be created to measure each aspect of the organization’s IT security. Selecting the most critical elements of the organization’s IT security program during metrics prioritization will make the program manageable and successful
14
IT security performance goals and objectives are expressed in the form of high level policies and requirements in many laws, regulations, policies, and guidance that describe the dimensions of an effective IT security program:
• Clinger Cohen Act
• Presidential Decision Directives 63
• Government Information Security Reform Act (GISRA)
• OMB Circular A-130, Appendix III
• Critical Elements withinNIST Special Publication 800-26
• Federal Information Security Compliance Audit Manual (FISCAM)
IT Security Performance Goals and Objectives
15
IT Security Policies, Guidance, and Procedures
Some Federal guidance and agency-specific policies and procedures provide more detailed information specific to the agency:
• NIST SP 800-12, 800-14
• Agency-specific policy and guidance
• Subordinate questions withinNIST Special Publication 800-26
16
System Security Program Implementation
System Security implementation includes:
• Processes and procedures in place
• Existing capabilities
• Areas for improvement
• Existing metrics
• Existing data sources that can be used to derive metrics data
These may be documented in the following sources:
• System Security Plans
• OMB Plan of Actions and Milestones (POA&M) reports
• Latest GAO and IG findings
• Tracking of security-related activities
• Risk assessments and penetration testing results
17
Metrics can describe three aspects of IT security program operations and management
7
Level of Implementation:Most organizations are new to measuring IT security with performance metrics. They will begin by measuring level of implementation of its security policies and procedures. Instituting a metrics program is the first step to process maturity.
Business Impactcan be measured through correlation analysis oncean organization’s processes are self-regenerating andmeasurement data gathering is transparent.
Security Program Effectiveness and Efficiency:As an organization’s process maturity increases and performance data becomes more readily available, metrics will focus on program efficiency and effectiveness.
18
It is important to record the specifics of each metric for the purposes of data analysis and possible metric reuse
Defines the metric by describing the quantitative measurement(s) provided by the metric
Describes the overall functionality obtained by collecting the metric
Specific questions that will need to be answered via survey or through automatic data gathering to be able to calculate the metric
Proposes the periods for collection of data to be used for measuring changes over time. Suggested time periods are based on likely updates occurring in the applicable process
Describes the calculation to be performed that results in a numeric expression of a metric
Lists the location of the data to be used in calculating the metric
Provides information about the meaning of the metric and its performance trend. Proposes possible causes of trends, identified through measurement, and points at possible solutions to correct observed shortcomings
Metric
Purpose
Survey Question(s)
Frequency
Formula
Target
Metric
Purpose
ImplementationEvidence
Frequency
Formula
Data Source
Indicators
19
Metrics can help identify causes of poor performance, including:
Insufficient human, monetary, or other resources can be causing negative performance trends
Lack of appropriate training for the personnel installing, administering, maintaining, or using the systems
Security patches that have been removed during the operating system upgrades
New or upgraded systems that are not configured with all required security settings and patches
Security patches or upgrades that are incompatible with software applications supported by the system
Lack of management awareness and/or commitment to security
Lack of policies and procedures that are required to ensure existence, use, and audit of required security functions
Poor system and security architectures that make systems vulnerable
Inefficient planning processes that influence the metrics (including communication processes necessary to direct organizational actions)
Resources
Training
System Upgrades
Configuration Management Practices
Software Compatibility
Awareness and Commitment
Policies and Procedures
Architectures
Inefficient Processes
Areas Examples
20
Employees sign employee agreements stating that they have read and understood rules of behavior
How does NIST SP 800-26 relate to metrics?Critical Element:Performance
Goal
PerformanceObjectives
Metrics
% of employees who signed employee agreements
% of employees who received annual refresher training
% of new employees who underwent security awareness training
Subordinate Questions:
. ..
13.1.4 Are methods employed to make employees aware ofsecurity, i.e., posters, booklets?
ImplementationEvidence
Rules of Behavior areincluded in training
Policies and Procedures discussionincluded in training
New employee training is conducted
Annual refresher training is conducted
13.1.1 Have employees received a copy ofthe Rules of Behavior?
13.1. Have employees received adequatetraining to fulfill their security responsibilities?
13.1.5 Have employeesreceived a copy ofor have easy access toagency security procedures and policies?
13.1.3 Is there a mandatory annual refresher training?
21
Example 1: Security Awareness, Training, and Education (Implementation)
Metric
Purpose
Data Source
Formula
Indicators
Percentage employees who underwent initial IT security awareness training
To determine the number of new employees who underwent required IT security awareness training, including receiving a copy of Rules of Behavior and security policies and procedures.
Semi-Annually
If response to 1-3 was “Yes” % = (# who took training) / (Total # of new employees)
Yes No 1. Is security awareness training required for new employees?
Critical Element 13.1 Have employees received adequate training to fulfill their security responsibilities?Subordinate Questions 13.1.1 Have employees received a copy of Rules of Behavior?13.1.5 Have employees received a copy of or have easy access to agency security procedures and policies?
Security awareness training is effective when it includes specific information on rules of behavior and information on security policies and procedures. This metric validates the content of training and determines the percentage of employees who took the training. High numbers close to 100% are highly desirable. Having employees who do not understand the security implications of their actions significantly increases the number of security incidents caused by insiders. This may also increase the opportunities for unauthorized access to systems caused by staff not following security policies and procedures.
Yes No 2. Do employees receive a copy of Rules of Behavior as a part of their training?
Yes No
3. Do employees receive a copy of policies and procedures or a summary with references as a part of their training?
4. How many new employees joined your organization during current past reporting period? Fill in the blank____________ How many of those took the training? Fill in the blank______________
Frequency
Security awareness training tracking by the responsible office, or manual or automated system.
ImplementationEvidence
22
Example 2: Incident Response Capability (Efficiency)
Metric
Purpose
Frequency
Formula
Data Source
Indicators
Elapsed time between when an incident is reported and when affected partiesare notified of a problem.
To determine how long it takes to notify users of incidents
Monthly
At the program, level average time = weighted average among all responses
Incident Response Database
Under 12
What is an average number of hours that it takes to notify users that may be affected by a reported incident?
Critical Element 14.1 Is there a capability to provide help to users when a security incident occurs in the system?Subordinate Question 14.1.3 Are incidents monitored and tracked until resolved?
Over 24
Users should be notified of a problem as quickly as possible. A delay in notification time may point to a lack of incident monitoring and tracking until resolution. Higher number points to inefficiency in (or lack of) a notification process. This metric can also be calculated as percentage of all incidents that are communicated to affected parties within 1 hour, 12 hours, 24 hours, and over 24 hours.
Under 1 Under 24
ImplementationEvidence
23
Example 3: Hardware and System Software Maintenance (Implementation)
Metric
Purpose
Frequency
Formula
Data Source
Indicators
Percentage of systems with latest patches installed
To quantify the level of risk exposure caused by the lack of current security patch implementation
Monthly
(# components with up-to-date patches)/(Total # of components)
Regular Vulnerability Scanning
1. Is regular vulnerability scanning conducted?
Critical Element 10.3. Are systems managed to reduce vulnerabilities? Subordinate Question 10.3.2 Are systems periodically reviewed for known vulnerabilities and software patches promptly installed?
Insufficient staff Superseding other priorities
This metric monitors compliance with applicable patches and provides useful information about the level of risk exposure at a system level. The goal in this case is 100%. The desired trend for this metric is downward. Question 3 identifies why a patch compliance validation process may be lacking and points at specific corrective actions to facilitate establishment of such process.
ImplementationEvidence
2. How many components are scanned every time? Fill in the blank ________3. How many of those had up-to-date patches during the last scanning cycle?
Fill in the blank ________
Yes No
4. If your answer to question 1 was “no”, what was the reason?
Insufficient funding
24
Metrics Development Criteria: What is a Good Metric?
Based on IT security performance goals and objectives: NIST SP 800-26 Critical Elements and Subordinate Questions are used to derive performance goals and objectives
Quantifiable: Metrics should yield quantitative rather than qualitative information to increase the objectivity and validity of data
Obtainable/Feasible to measure: Metrics data should be available or easily collected through interviewing or by accessing data repositories. If a metric requires significant modification of agency processes or implementing a new tool, data collection may not be feasible at this time
Repeatable: Measurements should be able to be repeated in a standard way at predetermined intervals to identify trends or identify if positive changes have occurred as a result of corrective actions
Provide relevant performance trends over time: Repeated measurements reveal change in a timely manner
Useful in tracking performance and directing resources: Metrics should be useful to stakeholders and should yield information that is important in financial decision making
26
Breakout Session
Goal: To complete a Metric Form for one of the metrics that is required for GISRA reporting for FY 2002. This includes identifying the NIST SP 800-26 Critical Element and Subordinate Question that map to the specific GISRA question from OMB guidance.
Duration: 30 minutes
Method:
• Read the metric your Breakout Group is assigned
• Select the NIST SP 800-26 Critical Element that includes your metric
• Select the Subordinate Question within the Critical Element that maps to your metric. Remember, a single metric can use more than one Subordinate Question
• Complete the Metric Form’s sections, giving particular attention to what implementation evidence may exist that corresponds to your Subordinate Question
Follow up: Each Group will have five minutes to brief their Formto the other groups. This brief should include:
• The metric your Breakout Group was assigned
• The Critical Element and Subordinate Question that maps to your metric.
• The completed Metric Form, including implementation evidence and indicators.A list of possible sources of the data you need to uncover for your metric
27
Metric
Purpose
Frequency
Formula
Data Source
Indicators
ImplementationEvidence
Critical Element:
Subordinate Question:
29
In this section, you will:
• Receive an introduction to the IT Security Metrics-related roles and responsibilities
• Learn the steps involved in IT Security Metrics program implementation by learning the process and following an example through the process
30
Multiple success factors can influence quality and sophistication of IT Security Metrics (slide 1 of 2)
Ensure that IT Security Metrics Program is manageable:
• Use no more than 10-20 metrics at a time, based on current priorities
• Phase old metrics out and phase new metrics in when performance targets are reached or when requirements change
Ensure acceptable quality of data:
• Data collection methods and data repositories should be standardized
• Events must be reported in a standard manner throughout the organization and the results of such reports need to be stored in the data repository
31
Multiple success factors can influence quality and sophistication of IT Security Metrics (slide 2 of 2)
Obtain organizational acceptance:
• Metrics need to be validated with organization’s stakeholders within headquarters and in the field
• Metrics should be vetted through appropriate approval channels
Ensure that metrics are useful and relevant:
• Useful data should be collected
• Not all data are useful
32
Metrics-related roles and responsibilities are dispersed throughout an organization
Responsibility forOrganizationalAcceptance ofMetrics Program
Responsibility forMetrics DataCollection andData Accuracy
33
Each organization will implement a metrics program specific to its needs
• Tailor to organization and business processes
• Identify IT Security Metrics-related stakeholder roles and responsibilities
• Lay out required infrastructure changes, such as creation of web-based data collection tools and of new data repositories
• Identify required modifications of the current data sources
• Define data reporting formats
34
IT Security Metrics data collection must be as transparent and non-intrusive as possible.
Output from standard security activities can be used to quantify IT security performance
• Incident Handling
• Testing
• Network Management
• Audit Logs
• Network and System Billing
• Configuration Management
• Contingency Planning
• Training
• Certification and Accreditation
35
IT Security Metrics Program Implementation Process
• Identify stakeholders• Determine goals /
objectives• Review existing metrics• Develop new metrics• Identify data collection
methods and tools• Collect metrics
• Analyze collected data• Conduct gap analysis
- Identify gaps between actual and desired performance
• Identify reasons for undesired results
• Identify areas requiringimprovement
• Determine range of corrective actions
• Select most appropriate corrective actions
• Prioritize corrective actions based on overall risk mitigation goals
• Develop cost model - Project cost for each
corrective action• Perform sensitivity analysis• Develop business case• Prepare budget submission
• Budget allocated• Available resources
prioritized• Resources assigned
• Management• Operational• Technical
• Track progress• Report as required
36
Process Implementation Example
Only 5% of employees receive annual IT security refresher training.
Lack of IT security refreshertraining may be causing weakpasswords, identified bya password cracker that isrun regularly.
Employees should be required to take annual IT security refresher training as part of their annual review process.
Since annual refresher training has ceased, the number of weak passwords has increased by 50%.
A budget submission detailingmetrics findings related to annual IT security refresher training was submitted, and funding received.
Annual refresher training,an operational control,is instituted.
Since the training was re-instituted, the percentageof weak passwords hasdecreased by 40%
37
Summary
• Discussed why Metrics are important for IT security
• Obtained understanding of the relationship between GISRA, NIST SP 800-26, and IT Security Metrics
• Described IT Security Metrics
• Described the Metrics Development Process
• Created metrics to be implemented at a system level through applying metrics development process
• Discussed metrics-related Roles and Responsibilities
• Described how to implement a Metrics Program
38
Next Steps
• You can immediately use what you have learned today to propose some metrics within your agency
• Notes of the workshop will be published in two weeks
• You can use the three metrics presented during the workshop and those that we developed together for your GISRA submission
• Metrics Guidance first draft will be published by September 30, 2002
• Please contact Marianne Swanson if you have any questions at [email protected], 301-975-3293