AT&T Heartbleed Response (OpenSSL Brief)

Adam Jones - CISM, CGEIT, CISSP, 6σGB

ITO Global Infrastructure Operations

July 24, 2014

© 2014 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change.

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.2

OpenSSL Zero Day Vulnerability

4/7 - Cert 720951 Issued for OpenSSL

4/7 - Cloudfare.com challenges internet to hack their keys. Two

participants reported success.

4/7 - Evidence of active attempts to exploit the vulnerability surfaced shortly after this event.

4/8 - CNET: "We were able to scrape a Yahoo username & password via the Heartbleed bug," tweeted Ronald Prins of security firm Fox-IT, showing a censored example. Added developer Scott Galloway, "Ok, ran my heartbleed script for 5 minutes, now have a list of 200 usernames and passwords for yahoo mail...TRIVIAL!"

4/21 - CNBC: “Obamacare enrollees urged to change passwords over Heartbleed bug”

Video Placeholder

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.3

Story Line

Startup Emerging Sustaining Response Close

Desktops - very low exposure

Network Elements - low exposure

UNIX/Linux hardware and application

processes had the majority of exposure

while modest given the overall enterprise.

This is the high level recap of AT&T’s OpenSSL Heartbleed critical response.

• Risk Review - Zero day alert issued.

• Evaluating exposure• Release

management processes begin testing and staging of available patches.

• SWAT mode• Processes confirm

some exposure.• Scanning processes

increased.• Reporting

enhancements.• Communication

plans commence.• Scans identify

hardware issue.

• Status change to standard operations.

• Communication plans continue.

• Social media in heavy usage internally.

• Update for hardware issue deployed.

• Final issues resolved.

• Patching wraps up.• Steps to update

certificates and passwords continue.

• Ongoing processes continue for any new hosts coming online.

4

Lessons Learned & Best Practices

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.5

Operational Recommendations - Lessons Learned

Inventory•Assets, Valid Owners, Hosted Application (Installed Applications), Application Contacts and Management (Business Unit association)

•Hostname, FQDN, IP Address, OS, OS Version, Patch Levels, Patch Date.

Communication Plans•Delivery - Application Contacts, Operations contacts (SA, DBA, Supervising Managers)•Executives - SA, DBA Executives, Application Executives.•Social Media - Strongly encourages for larger enterprise environments.

Reporting•Recommended - focus on open database relationships, common primary and secondary keys, databases of applications and each application having current accurate relations to core inventory.

•Online reports should be intuitive and actionable. Export functionality with pivot table structures is recommended for increased productivity.

Release Management•Critical - This is imperative for availability and rapid remediation. Mature processes for testing and certifying release packages prior to distribution is pivotal to success.

•Best practice a core functional set of teams, favorable is strong processes with cross functional teams.

Layered Security•Cyber defense as it is well documented is based on layered security controls. Rapid remediation and or containment is dependent on multiple controls working in harmony (IDS, IPS, Scans, Patch Management, Reporting, etc.).

Slide 4

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.6

Reporting Logical View

Server Tracking (Unique Servers)

Response Tracking

Summary Views

Detailed Scheduling

Application Level Tracking

Response Submissions

Detailed views for compliance,

scheduling

Evidence (Patched vs Non Patched)

App Risk - Direct and Shared Compliance and Risk Views

Servers

Apps

Results

Operations Systems Level Data• OS views per platform, what is compliant vs planned

vs documented.• Client response interfaces for organizing what clients

have sent in and which apps have not sent responses.• Real time audit trail.

Clients App Towers (Hosted Applications)• Automated Communications - App Teams.• Data driven reports for GM Communications.• App Instance Tracking.• Interfaces for reconciling response for questions,

scheduling requests and jeopardy submissions.

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.7

Communications Best Practice

1. Audience - Inventory dynamically feeds automation that sends this message to the correct audience once triggered. Target users are application contacts using impacted servers.

2. Media Types - Use multiple media forms in one communication (i.e. email, slide deck, video overviews).

3. Reference Material - Have mature reference areas available (wiki, social media site, any online reporting sites, video references).

4. Required Action - Must include clear, actionable steps. No communication will be 100% successful but the steps have to be very meaningful.

* Recommendations are based on standard processes for internal operations.

8

Feedback, Questions

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.9

Questions and FeedbackAudience - Our team would like your input and questions.

Scanning• How is your company scanning full ip address ranges for all ports

internal and external?

Inventory • IP Address reconciliation - How is your company managing

unknown IP Addresses that do not map to a known owner?

Reporting• How is your company discovering non Microsoft platforms? • How standard is your environment?

Application Availability• How does your company maintain availability of your enterprise

applications while expediting emergency changes into the environment?

• Change control?• Standard change windows?• Testing?

• Certificates - How did you handle these changes?

Social Media• Is your company using social media to collaborate on security

remediation efforts?

References:

• ISACA Incident Management and Response• http://

www.isaca.org/Knowledge-Center/Research/ResearchDeliverables/Pages/Incident-Management-and-Response.aspx

• ISACA Security Incident Management Audit/Assurance Program • http://

www.isaca.org/Knowledge-Center/Research/ResearchDeliverables/Pages/Security-Incident-Management-Audit-Assurance-Program.aspx

• AT&T ThreatTraq Spotlight• http://

techchannel.att.com/play-video.cfm/2014/4/9/AT&T-ThreatTraq-Spotlight-Heartbleed

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.10

Thank YouAdam Jones - CISM, CGEIT, CISSP, 6σGBSr. Technical Team LeadAT&T Global Infrastructure Operations

Office: 478-461-3070Email: adam.jones@att.comLinkedIn: https://www.linkedin.com/in/ajones07

Rebecca Finnin - CIPP, CISSP, CISA, CPADirector AT&T Chief Security Office

Email: rebecca.finnin@att.com