money$ec evolved
DESCRIPTION
Money$ec Evolved. Wherein not everything has a tidy baseball analogy. Jared Pfost Chief Executive Officer Third Defense. Brian Keefer Security Architect Leading SaaS Security Company. Recap. Last year we applied baseball “SABRmetrics” to InfoSec We spent some time in the real world - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/1.jpg)
Money$ec EvolvedWherein not everything has a tidy baseball
analogyJared PfostChief Executive OfficerThird Defense
Brian KeeferSecurity ArchitectLeading SaaS Security Company
![Page 2: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/2.jpg)
Recap•Last year we applied baseball
“SABRmetrics” to InfoSec•We spent some time in the real
world•Oh yeah, some guy named Brad
was in a movie
![Page 3: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/3.jpg)
In case you missed it
How Analytics Changed Baseball
![Page 4: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/4.jpg)
Oakland A’s•Teams bid for players in Free Agent
market•Start of 2002 A’s had payroll
~$40M*•NY Yankees payroll ~$126M*•So poor teams have no shot at
winning, right?*From “Moneyball”
![Page 5: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/5.jpg)
1999-2001Team Wins Losses Est
Payroll*
NYY 280 203 $257M
OAK 280 205 $70M
*Estimate from baseball-reference.com
![Page 6: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/6.jpg)
Billy Beane•GM Billy Beane
defied convention
• i.e. he didn’t follow “best practices”
•made data-drive decisions
•Hired Paul DePodesta
![Page 7: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/7.jpg)
Traditional baseball•Talent is evaluated by scouts•Scouts are usually washed-up
players•i.e. “Industry veterans” or
“experts”•Value statements are largely
subjective
![Page 8: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/8.jpg)
Next-gen Baseball
•Started in 1977•Bill James wanted to see what
influenced game outcome•Realized stats created in 1859
didn’t properly attribute events
![Page 9: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/9.jpg)
Key lessons•Don’t make emotional decisions•At least recognize your bias
•Collect the “right” data•Look for correlations
•Set reasonable criteria for success•Don’t overspend
![Page 10: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/10.jpg)
This Applies to InfoSec
![Page 11: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/11.jpg)
Problem statement
•Every organization is competing with attackers
•Most don’t have Fortune 50 budget•How can you be effective?
![Page 12: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/12.jpg)
Conventional “wisdom”
•“Everyone knows” that you need•Firewall•Anti-virus•Change passwords frequently•Prohibit social networking•Etc.
![Page 13: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/13.jpg)
Do they work?
•Port 80 goes through the firewall•Anti-virus misses custom malware•Stolen passwords used quickly•Social networking key to marketing
and employee satisfaction
![Page 14: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/14.jpg)
Clearly this is not working
•Do we actually want a new strategy?
•What does winning look like?•How do we get started?
![Page 15: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/15.jpg)
Cheap & Easy
Spend to Comply
Fix Gaps Now!
Ok, how much do we really need...?
Are You Ready To Win?
Motivating Event
![Page 16: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/16.jpg)
•Winning is not losing...•No unacceptable risks realized •Cheap as possible
What Does Winning Look Like?
![Page 17: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/17.jpg)
So, about that...• Started collecting info• Realized it was far from
complete• Historical incident
rates were meaningless
• Minimal ability to measure what helps
• 12 metricsMoney$ec 1.0
![Page 18: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/18.jpg)
EvolutionMoney$ec 2.0
• Measure what’s easy
• Set Targets• Justify More• Optimize
Cost vs. Target
![Page 19: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/19.jpg)
Start With “Easy”• Incidents - # of High, Moderate, Annoying• Application- # of Post-production application bugs
• Passwords- % passwords easily guessed• Scanned Vulnerabilities- # Patch & config vulns not mitigated per Severity Service
Level
![Page 20: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/20.jpg)
Real Metrics Have Outcomes
• Stats are trendy, Metrics have Winners|Losers–Measure actual performance against target–Benefits
• Drives “acceptable risk” conversation with Management• Simplifies reporting e.g. are we above|below?
![Page 21: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/21.jpg)
Back To “Easy”• Scanned Vulnerabilities
- # Patch & config vulns not mitigated per Severity Service Level- Sev 1 Server Vulns Mitigated within 30 days- Sev 2 within 60 days
![Page 22: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/22.jpg)
You really can do this
![Page 23: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/23.jpg)
Ooooh, shiny!
![Page 24: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/24.jpg)
24
Expand Measurement• Access Management
- % Employee termination within policy- % Role/Access verification• Network- % critical systems monitored- Moving to % of full packet capture
• Vendors- % assessed per policy- # overdue findings• Employee- # of duplicate incidents• Change Management- # emergency or unplanned changes- % of changes with a regression
Every Metric Must Have A
Target
![Page 25: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/25.jpg)
Optimize Cost - Target•Is target too high?
67
75
84
92
100
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb
Proposed Target
![Page 26: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/26.jpg)
Cost - Benefit - AccountabilityRate Hrs Per
Test/Deploy# Personnel Cost Per
Server Update
$100/HR 40 10 $40,000
Evidence: Incidents, response performance, attack attempts
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10
DoS PostMalware Post
Worm Post
Or
http://code.google.com/p/openpert/
Current Target
Proposed Target
![Page 27: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/27.jpg)
Improve IR•Move IR out of IT?•Infections are incidents•Data is needed to evaluate
controls•Knowing root-cause guides future
controls and Targets
![Page 28: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/28.jpg)
Integrate Metrics Into Root Cause Analysis
Find Leading Indicators
![Page 29: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/29.jpg)
Parting Thought
•People implicitly decide not to measure.
•Money$ec says explicitly decide when you don’t.
![Page 30: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/30.jpg)
Security Reformation?
http://www.liquidmatrix.org/blog/2012/02/21/we-are-losing/
http://lifecypha.wordpress.com/
![Page 31: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/31.jpg)
Time to Share
•Data you find useful to collect?•Spotted any correlations?•Proved any controls too expensive?•What communities do you
participate in?
![Page 32: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/32.jpg)
Thanks!
Brian Keeferb: http://rants.effu.se
e: [email protected]: @chort0
Jared Pfostb: http://thirddefense.wordpress.com
e: [email protected]: @JaredPfost
![Page 33: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/33.jpg)
appendix
![Page 34: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/34.jpg)
Task InfoSec Control Owner Business Owner
Define Metric A,R R C
Define Target R R A,R
Report Metric A,R R I
Review Target A,R R R
R – ResponsibleA – AccountableC – Contribute
I - Informed(There can be only one “A”)
RACI in action
![Page 35: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/35.jpg)
2011 VZ DBIR vs. Money$ec
![Page 36: Money$ec Evolved](https://reader036.vdocuments.site/reader036/viewer/2022062501/56815f2f550346895dcdfbfd/html5/thumbnails/36.jpg)
Device Patch & Config Monitoring