moneysec - moneyball for security
DESCRIPTION
Making Better Security Investments - CIO Perspectives Atlanta 2013TRANSCRIPT
![Page 1: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/1.jpg)
1
Phil AgcaoiliWednesday March 13, 2013
Moneysec - Moneyballfor Information Security
![Page 2: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/2.jpg)
2
1st: Moneyball Overview2nd: Introduction to Moneysec3rd: Evidence-based Data for MoneysecHome Run: What can you do with this?
Agenda
Jared PfostChief Executive OfficerThird Defense
Brian KeeferSecurity ArchitectLeading SaaS Security Company
Moneysec ideas borrowed from:
![Page 3: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/3.jpg)
3
Moneyball
• Reshaping a laggard team into a world-class winner–with one of the lowest budgets
• “Willingness to re-think baseball: how it is managed, how it is played, who is best suited to play it, and why. “– Michael Lewis, author– Which statistics are most correlated to winning games – Focus on these success metrics and actually use them for
recruiting, player development, and game-time decisions.– Full management commitment
![Page 4: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/4.jpg)
4
Car V
alue
Car Body Color Car Insurance Cost#
of C
ar A
ccid
ents
Car F
uel E
ffici
ency
Car Mileage
negative zero positive
The degree that two measurements or variables show a tendency to vary together
Correlation
![Page 5: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/5.jpg)
5
(hits + walks) x total basesruns created =
(at bats + walks)
The Moneyball Formula
Source: Michael Lewis - Moneyball
Metric: On-base Percentage instead of Batting Average
Bill James’ formula:
“…it implied, specifically, that (professional baseball people) didn’t place enough value on walks and extra base hits . . . And placed too much value on batting average and stolen bases.”
“…The details of James’s equation didn’t matter all that much…What mattered was (a) it was a rational, testable hypothesis; and (b) James made it so clear and interesting that it provoked a lot of intelligent people to join the conversation.” p. 78
![Page 6: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/6.jpg)
6
Offensive strategy
Source: Michael Richmond
Defensive strategy
![Page 7: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/7.jpg)
7
(hits + walks) x total basesruns created =
(at bats + walks)
Moneyball Formula in Action
(1,556 + 571) x 2,360runs created = = 794
(5,635 + 571)
2011
787 actual
![Page 8: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/8.jpg)
8
Team - 2011 At Bats Hits WalksTotal Bases
Actual Runs
Predicted Runs Variance
% Variance
Texas Rangers 5,635 1,556 511 2,360 787 794 (7) -1%Kansas City Royals 5,604 1,534 471 2,238 676 739 (63) -8%Minnesota Twins 5,568 1,521 559 2,347 781 797 (16) -2%Cincinnati Reds 5,579 1,515 522 2,432 790 812 (22) -3%Detroit Tigers 5,643 1,515 546 2,343 751 780 (29) -4%Boston Red Sox 5,646 1,511 587 2,546 818 857 (39) -5%Chicago White Sox 5,484 1,467 467 2,303 752 748 4 0%New York Yankees 5,567 1,485 662 2,427 859 837 22 3%St. Louis Cardinals 5,542 1,456 541 2,227 736 731 5 1%Colorado Rockies 5,530 1,452 585 2,349 770 782 (12) -2%Milwaukee Brewers 5,606 1,471 546 2,376 750 779 (29) -4%Philadelphia Phillies 5,581 1,451 560 2,307 772 755 17 2%Baltimore Orioles 5,554 1,440 424 2,145 613 669 (56) -8%Atlanta Braves 5,463 1,411 634 2,190 738 735 3 0%San Francisco Giants 5,488 1,411 487 2,241 697 712 (15) -2%Chicago Cubs 5,512 1,414 479 2,213 685 699 (14) -2%Oakland Athletics 5,448 1,396 527 2,059 663 663 0 0%Florida Marlins 5,531 1,403 514 2,227 719 706 13 2%Los Angeles Dodgers 5,426 1,368 533 2,056 667 656 11 2%Washington Nationals 5,418 1,355 503 2,114 655 663 (8) -1%Arizona Diamondbacks 5,473 1,366 589 2,275 713 734 (21) -3%New York Mets 5,465 1,361 502 2,091 656 653 3 0%Los Angeles Angels 5,488 1,363 466 2,142 681 658 23 3%Toronto Blue Jays 5,495 1,364 471 2,496 755 768 (13) -2%Cleveland Indians 5,487 1,362 545 2,076 646 656 (10) -2%Houston Astros 5,452 1,348 415 1,974 611 593 18 3%Tampa Bay Rays 5,439 1,343 672 2,192 802 723 79 11%San Diego Padres 5,434 1,338 538 2,018 665 634 31 5%Pittsburgh Pirates 5,386 1,303 463 2,011 587 607 (20) -3%Seattle Mariners 5,409 1,274 459 1,836 513 542 (29) -5%
Average Variance -1%
![Page 9: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/9.jpg)
9
Oakland A’s• Teams bid for players in Free Agent market• Start of 2002 A’s had payroll ~$40M*• NY Yankees payroll ~$126M*• So poor teams have no shot at winning, right?
*From “Moneyball”
Team Wins Losses Est Payroll**
NYY 280 203 $257M
OAK 280 205 $70M
**Estimate from baseball-reference.com
1999-2001
![Page 10: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/10.jpg)
10
Billy Beane• GM Billy Beane defied convention
– He didn’t follow “best practices”– He made data-drive decisions– Hired Paul DePodesta
“The evaluation of young baseball players had been taken out of the hands of old baseball men and placed in the hands of people who had what Billy valued most . . . a degree in something other than baseball.” p. 41
“What you don’t do is what the Yankees do. If we do . . . We lose every time, because they’re doing it with 3 times more money than we are . . . The poor team was forced to find bargains: young players and whatever older guys the market had undervalued.” p. 119
![Page 11: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/11.jpg)
11
Traditional Baseball
• Talent is evaluated by scouts• Scouts are usually washed-up players• i.e. “Industry veterans” or “experts”• Value statements are largely subjective
![Page 12: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/12.jpg)
12
A few word about scouts…
Should we say outdated?
![Page 13: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/13.jpg)
13
Next-gen Baseball
• Started in 1977• Bill James wanted to see what influenced
game outcome– Realized stats created in 1859 didn’t properly
attribute events
![Page 14: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/14.jpg)
14
Key lessons
• Don’t make emotional decisions– Recognize your bias
• Collect the “right” data– Look for correlations
• Set reasonable criteria for success– Don’t overspend
![Page 15: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/15.jpg)
15
Moneyball Metrics
• Not all measures are equally important (80/20)• Watch out for “analysis paralysis”.• What are the most meaningful measures?• Less is more. Allows focus.• What processes are intuitively managed that could
be better run based on statistical facts?• How would this affect your culture and how you hire,
develop, promote, and field a security team based on these insights?
![Page 16: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/16.jpg)
16
Moneyball Metrics• Key to the value of Moneyball Metrics
True potential = not just measure, but also:– Track and trend performance over time– Benchmark performance vs. self (and peers)– Identify strengths and weaknesses– Diagnose - understand the interrelationships and
underlying drivers of performance– Prescribe actions to improve performance– Establish performance goals for both individuals
and overall security team– Become “World-Class”All metrics are worthless – unless you do
something with them.
![Page 17: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/17.jpg)
17
What can we learn fromMoneyball?
“Why didn’t anybody do this before?”
MoneysecA practical approach to
security investments
![Page 18: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/18.jpg)
18
Throw Money at the Security Problem?
![Page 19: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/19.jpg)
Problem Statement
• Every organization is competing with attackers• Most don’t have Fortune 50 budget• How can you be effective?
![Page 20: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/20.jpg)
20
Conventional “Wisdom”
• “Everyone knows” that you need– Firewall– Anti-virus– Change passwords frequently– Prohibit social networking– Etc.
![Page 21: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/21.jpg)
21
Do they work?
• Port 80 goes through the firewall• Anti-virus misses custom malware• Stolen passwords used quickly• Social networking key to marketing and
employee satisfaction
![Page 22: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/22.jpg)
22
Clearly this is not working
• Do we actually want a new strategy?• What does winning look like?• How do we get started?
• Applying new model• Use the security stats that are out there
– Verizon Data Breach Investigations Report– Trustwave Global Security Report– Ponemon Institute Cost of Data Breach Report and Research Studies– Manidant M-Trends Report– Symantec reports– CSO Magazine Global State of Information Security Survey– Metricon
![Page 23: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/23.jpg)
23
Models for Research and Investment
• Inductive– starts with data available– concludes with possible
hypotheses– bottom up data driven
approach
• Deductive– starts with theoretical
framework– concludes with logical
deductions– theory driven approach
data
interpretation
theory development
hypothesis testing
hypothesis
theory
![Page 24: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/24.jpg)
24
TrendsFix what’s broken• Hacks and compromise
– Fix what’s already been hacked at your company• Understand security trends for your industry
– Small and Medium Business beware– Banks – DDOS, fraud, botnets, and web authentication attacks– Hospitality – Credit cards, point of sale systems, Wifi, and admin
accounts– DIB – RSA hack, Adobe/Microsoft 0days, remote access, and phishing– News – NYT/WSJ, phishing, Oracle Java 0days– Retail – Open Wifi, POS– LEA – 0day, social engineering and phishing– Credit card processors – Phishing and egress traffic– Websites – Sony (SQL Injection) and exclusion from core security
• Evaluate your threat landscape to prioritize your treatment strategy
MotivatingEvent
![Page 25: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/25.jpg)
25
2012 Verizon Data Breach Investigations Report (DBIR)
• 5th year of public releases– Starting in 2008– 7 total reports (mid-year
supplementals in 2008 and 2009)
• Dataset now contains:– 8 years of data
![Page 26: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/26.jpg)
26
2012 Verizon Data Breach Investigations Report (DBIR)
2012 Trustwave Global Security ReportIn those cases in which an external entity was necessary for detection, analysis found that attackers had an average of 173.5 days within the victim’s environment before detection occurred.Conversely, organizations that relied on self-detection were able to identify attackers within their systems an average of 43 days after initial compromise.
![Page 27: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/27.jpg)
27
2012 Verizon Data Breach Investigations Report (DBIR)
![Page 28: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/28.jpg)
28
2011 Verizon Data Breach Investigations Report (DBIR)
![Page 29: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/29.jpg)
29
Who are the (external) bad guys?
• Eastern Europe takes a commanding lead
2011 Verizon Data Breach Investigations Report (DBIR)
![Page 30: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/30.jpg)
30
Who are the (internal) bad guys?
• Quite a jump in regular users (was 51% last year)
• % of breaches involving Finance staff doubled
• % of breaches involving executives increased from 7% to 11%
2011 Verizon Data Breach Investigations Report (DBIR)
![Page 31: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/31.jpg)
31
Customized Malware
2011 Verizon Data Breach Investigations Report (DBIR)
![Page 32: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/32.jpg)
32
Hacking Methodologies
2011 Verizon Data Breach Investigations Report (DBIR)
![Page 33: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/33.jpg)
33
Social Engineering and Physical Security Trends
• 11% of breaches employed some level of social engineering (down from 28% last year)
2011 Verizon Data Breach Investigations Report (DBIR)
![Page 34: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/34.jpg)
34
Device Patch & Config Monitoring
2011 VZ Data Breach Investigations Report and Moneysec
![Page 35: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/35.jpg)
VZ DBIR and Trustwave GSR
2012 Verizon DBIR
2012 Trustwave GSR
![Page 36: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/36.jpg)
36
Confidential Documents at Risk StudyKey Findings• The negligent insider seems to pose the greatest risk because of poor
internal controls and improper accessing and transferring confidential documents.
• Sensitive documents are most at risk at the document and file level.• Governance tasks or procedures for privilege and access to sensitive
documents need improvement.• Budget and compliance monitoring procedures are the critical success
factors to achieving good internal controls and governance procedures.
![Page 37: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/37.jpg)
37
Information Leakage
• Ex-employees, partners, and customers• Over 1/3 due to negligence• Increasing loss from external collaboration
Percentage cause of data breach
Cost of Data Breach reportPonemon Institute 2010
Estimated sources of data breach
2010 CSOGlobal State of Information Security Survey
![Page 38: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/38.jpg)
38
Evidence-based Investments• Don’t protect everything
– Protect most important data and services• Small, targeted investments
– Pass the Red Face Test – Reduce Investments through integration• Antivirus - Forefront• Full Disk Encryption – Bitlocker
– Patch and harden configs – Change default credentials and restrict/monitor privileged accounts– Increase focus on closing the detection and response gap– Secure development through application testing and code reviews– Increase awareness and change culture
• Social engineering and phishing• Destroy what you don’t need
• Treat all endpoints as hostile• Collapse to cores
– Protect cores really, really well• Collect your own metrics and apply security as necessary
![Page 39: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/39.jpg)
39
Moneysec Metrics • You can measure anything! Even intangibles.• You don’t always need to be exact.• Reducing uncertainty adds value.• Having just some data can go a long way to help a decision maker.
• Use industry data – You’re not a beautiful snowflake– Apply what correlates
• Moneysec metrics– Measure what’s easy– Set targets– Justify as needed– Optimize Cost vs. Target
• More metrics:– Moneysec Evolved– Metricon
Source: Douglas Hubbard
![Page 40: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/40.jpg)
40
ConclusionQuick attempt to grossly oversimplify the theme: •Billy Beane developed new metrics to win for less.•Use industry security performance data to help prioritize spending to win for less.•Internal information security metrics can be applied along with industry security performance data.•Measure well. Gather evidence.•Determine if you need more or less.•Apply inductive and deductive approaches to make better investments.•Use the evidence you have for better investments.
![Page 41: Moneysec - Moneyball for Security](https://reader035.vdocuments.site/reader035/viewer/2022062405/5577daefd8b42a7b7b8b46e9/html5/thumbnails/41.jpg)
41
Questions & Answers
Phil AgcaoiliCo-Founder & Board Member, Southern CISO Security CouncilDistinguished Fellow and Fellows Chairman, Ponemon Institute Co-Chair, Communication Sector Coordinating Council (CSCC),
Cybersecurity Committee – Technology Sub-CommitteeFounding Member, Cloud Security Alliance (CSA)Inventor & Co-Author, CSA Cloud Controls Matrix, GRC Stack,
Security, Trust and Assurance Registry (STAR), and Open Certification Framework (OCF)
@hacksec https://www.linkedin.com/in/philA