application assessment metrics
DESCRIPTION
Presentation by Yvette du Toit at ISSA in 2011. This presentation is about application assessment metrics and their challenges. Examples of Sensepost metrics are given.TRANSCRIPT
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
PRESENTED BY: Yvette du Toit
Agenda!
• Background!• Approach!• Examples!• Challenges with Application Security Metrics!• Q&A!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Background!
• As Security Consultants we write reports!– Test, analyse, write up findings, submit to client!
• Issues still remain open – why?!– Reports not say enough!– Question value report offer!
• Solution – metrics / visualisation!– Graphs, colour, size etc!
• First – letʼs take a look at what reports say…!– Qualitative ratings!– Best practice!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
What do Reports Say?!
• 2007 - 2011!• Many words….!• Content (Exec Summary, Technical Summary, Conclusion)!• Are actions effective?!• What would be more valuable – comparison (time & peers)!• How do we use metrics?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Pages Words Assessments 638 224587 Re-Tests 137 28164 Total 775 252751
Approach!
• Metrics – definition!– Quantifiable!– Characteristics!
• 3 Metric Veterans:!– Jacquith - “those that support decision making about risk for the
purpose of managing that risk” !– Marty – “a picture paints a thousand log records”!– Godin: “just because something is easy to measure doesnʼt mean
itʼs important”!• NB: To measure what is important & that will yield “useful”
information!– Examples of metrics not necessarily useful!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Definition!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Example!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Metrics can be misleading!
Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Example!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Metrics are not always 100% useful!
Approach!
• Why?: illustrate useful information!– Recurring issues!– Time required to compromise!– Top 10 list!– Effectiveness of remediation!– Benchmarking!
• Who? 7 organisations in financial sector!• When? 3 ½ years!• How? Data capture process!
– Marco Slaviero (Head of R&D)!– Spreadsheet for data capture!– Report meta-data (project length, frameworks, dates etc.)!– Findings categorised (pre-defined list of vulns)!– Findings ranked (Impact, EoE, Threat metric)!
• Normalisation !– Allows for comparison across time and peers !
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Introduction!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Annual Distribution of Project (Days)!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics Proposal!
• Metrics extracted from report data:!– Timelines (plotting projects on timeline)!– Basic counts and statistics (uncover counts)!
• Number of projects!• Number of days!• Number of words and pages in report!
– Threat metrics (Findings per threat level)!– Bug class metrics (Findings across categories) !– Top 10 list !– Re-Test Metrics!– Benchmarks (comparison to peers)!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Our Metrics!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Timelines!
• Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
! "#$%&! '()*&!
!"#$%&'()'*++%++#%,-+' ./0' 112304'
!"#$%&'()'5%67%+-+' 8/4' 108.2'
7(-9:' 443' 131438'
SensePost Metrics in Action: Threat Metrics!
• Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Bug Classes!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful?!• See 56% of findings occur in Top 11 bug classes!• 2008 Anomaly (No Re-Tests) !
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Top 10!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful? !• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Re-Test!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful?!• 29% Critical and 42% High-risk issues remain open !
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Benchmarks!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful?!• Our client positioned 3rd (not highlighted here)!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Challenges!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Bug counts vs bug classes!– Bug counts – number of findings!– Bug classes – categories!– 2 applications scenario (10 findings 1 bug class vs 1 finding in 10 bug classes)!
• Depth vs breadth!– Each occurrence – depth!– Each bug class - breadth!
Q&A!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Thank you!• Longer paper – mail me!• Email: [email protected]!• Contact: +27 79 509 8913!