edge 2014: bypass surgery - akamai's heartbleed response case study
DESCRIPTION
Bypass Surgery - Akamai's Heartbleed Response Case Study by Brian Sniffen, Chief Security Architect, Akamai Technologies In April of this year, the inevitable happened and Akamai's network was determined to be vulnerable to the Heartbleed bug. The practice of information security is both about preventing vulnerabilities and mitigating vulnerabilities when they're found. In this case study, Akamai Chief Security Architect Brian Sniffen will walk through Akamai's response to the Heartbleed vulnerability and provide insights into the lessons to be learned for improved security processes and incident response. Akamai Edge is the premier event for Internet innovators, tech professionals and online business pioneers who together are forging a Faster Forward World. At Edge, the architects, experts and implementers of the most innovative global online businesses gather face-to-face for an invaluable three days of sharing, learning and together pushing the limits of the Faster Forward World. Learn more at: http://www.akamai.com/edgeTRANSCRIPT
Bypass Surgery and Other Tales Brian Sniffen
©2014 AKAMAI | FASTER FORWARDTM
Akamai Security Research & Architecture
• Crypto engineering expertise • Technical backstop • Product review • Akamai Architecture Group seat • Safety engineering • Incident management
©2014 AKAMAI | FASTER FORWARDTM
2014: The Year of Vulnerabilities
“You people in InfoSec have become the Product Managers!”
Yes, and we can’t wait to get out of that role.
• Heartbleed • INRIA-Prosecco Cookies • Shellshock
2009 2010 2011 2012 2013 2014
1 0 0 1 1 3+
2008
1
©2014 AKAMAI | FASTER FORWARDTM
Akamai Incident Management Principles
• Technical Incident Manager (TIM) coordinates all work • Incident Executive communicates upwards, shields TIM • GSS Business Incident Leads manage customer comms
• No single-point harm can cause a Severity 1 Incident • A hot meal and 6 hours sleep fix more problems than an all-nighter • If the TIM becomes an SME, get a new TIM
©2014 AKAMAI | FASTER FORWARDTM
We tell ourselves who we wish to be:
• Akamai says thank you. • Akamai doesn't respond to name calling,
but does respond to the useful technical content. • Akamai presents itself as a responsible and respected member
of the Internet community. • Akamai will use this incident to improve both its own security
and the general security of the Internet. • Akamai can laugh at itself.
©2014 AKAMAI | FASTER FORWARDTM
Heartbleed mail
From: Brian Sniffen Date: 7 Apr 2014 21:34:08 Subject: Sev 1: Heartbleed Will, I'm contacting you because you're the Ghost SME on call. I'm looking for evidence to refute the statement: "The Heartbleed bug can’t extract long-term customer private keys from a Ghost; we put them only in a wired, mmaped page way lower on the stack."
©2014 AKAMAI | FASTER FORWARDTM
Heartbleed Timeline
April 1: Notice; QA begins April 4: last Akamai Deployed Systems patched April 7, 1pm: Public Notice April 7, 6pm: What did we leak? April 8, 1am: Working exploit in Akamai lab April 9–12: Hastily publish Akamai Secure Allocator April 13, 11pm: Begin cert rotations & revocations
©2014 AKAMAI | FASTER FORWARDTM
“Don’t worry, we restored the old functionality”
April 14, 6am: “Why is this message in the old log format?”
A “Manual Change” had restored an old version.
©2014 AKAMAI | FASTER FORWARDTM
The Akamai Secure Memory Allocator
• 1999 code • One author, three redactors • State machine inspired by CLOS “advice” system
Turns out it works fine
Code Secure Heap mmap’d file Long-term
Allocations Heap
©2014 AKAMAI | FASTER FORWARDTM
Cert Revocation Progress
21 Apr
28 Apr
5 May
12 May
19 May
26 May
2 Jun
9 Jun
70% 90% 95%
# of
cer
ts
16 Jun
23 Jun
30 Jun
©2014 AKAMAI | FASTER FORWARDTM
Learning from Heartbleed
Nobody’s paying for OpenSSL! Practice in mass, fast, patching Practice in releasing helpful patches Simplicity promotes safety.
©2014 AKAMAI | FASTER FORWARDTM
Shellshock Timeline
Sep 23, 12pm: Notice from Florian Weimer, Debian Security Sep 23, 9pm: Manual change: replace bash wish dash;
Patches started Sep 24, 5am: WAF rule in place
SSH command= systems made safe Sep 24, 12pm: Public Notice Sep 25: “Kobrin Patch” to remove dangerous feature Sep 28: bash mostly replaced with dash on deployed network
©2014 AKAMAI | FASTER FORWARDTM
Bash patches
Pre-release: • Embargoed patch: 195 lines, 7 files (1/6 CVEs) • Kobrin patch: 2 lines, 1 file (6/6 CVEs)
Post-release: • NetBSD patch: 3 lines, 2 files (6/6 CVEs) • Fixed patch: 164 lines, 11 files (6/6 CVEs) • Apple patch: unpublished (exposure unclear)
©2014 AKAMAI | FASTER FORWARDTM
SSH command= limits
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFnHfYyS5onAN72oFpaopm+/yKbRy/TCwpt7Tmw3lk0P [email protected] command="/a/bin/akamai_run suspend" ssh-ed25519 AAAAB3NzaC1yc2EAAAADAQABAAABAQDKVmNk8leXjKkWZUHQjJITzrX+n1aa1xfBwK9Yp42q [email protected] V=“() { :;}; /bin/bash” ssh example.com :
©2014 AKAMAI | FASTER FORWARDTM
Akamai Shellshock exposures
sshClient Authgate Server 1
Server 2
Server 3PerforceWeb KerberosCGI
sshhttps
©2014 AKAMAI | FASTER FORWARDTM
Solaris 10
“We don’t have any Solaris 10 admins”
©2014 AKAMAI | FASTER FORWARDTM
Who’s looking?
13000 IPs probing per day
©2014 AKAMAI | FASTER FORWARDTM
Learnings from Shellshock
Nobody’s paying for Bash. And it was written in the 1980s! Simplicity promotes safety.
©2014 AKAMAI | FASTER FORWARDTM
The New Normal
• Two or three internet-wide patching incidents per year • Enterprise-wide compliance takes months • Trust less code. • Trust code less. • Treat upstream code like you wrote it?
• Homework: set up 24/7 contacts and Security contacts