duncan - what we learn from cyber exercises or not

2007 FIRST Annual Technical Conference – Sevilla, España

What We Learn fromCyber Exercises,

or NotJim Duncan

CSIRT Coordinator, BB&T2007 June 20

Duncan - What We Learn from Cyber Exercises, or Not2007 FIRST Annual Technical Conference – Sevilla, España 2

Overview

• Background• Purpose of exercises• Examples of what we can learn…• And what we fail to learn (repeatedly)• Purpose of exercises, redux• Future improvements• What else?


Background

• General cyber-security expertise with a special focus on incident response

• Product security work as well as critical infrastructure protection issues (ISACs)

• Varying amounts of involvement with many different cyber exercises including Cyber Storm, Livewire, various ISACs…

• And many real disasters, too• Exercise details not included


Why Conduct Exercises?

• So we know what to expect & what to do!▪ People in disasters fall into 3 categories:

• 10% -- 15% remain calm & act quickly• 15% or less COMPLETELY FREAK OUT!• Remainder are “stunned and bewildered”.

[John Leach in Aviation, Space, and Environmental Medicine, 2004]

▪ Survivors anticipate & plan accordingly▪ Do you review the safety

card every time you fly?


What Can We Learn?

• How will we react?• Who will be the real stakeholders?• What capabilities will succeed or fail?• What are the unforeseen obstacles?• What serendipity awaits us?• What better estimates can we calculate for

cost-benefit analyses?


How Will We React?

• Perhaps the most obvious goal is to test an organization’s response to a crisis

• When handling new information, the brain slows down (e.g.,1977 Tenerife accident)

• Under stress, it slows down even more! 45% of people “shut down” in a crisis

• Minimize “milling”; time is very valuable• Mitigate “disbelief”; Act now!


Who Will Be the Real Stakeholders?

• A critical point in the development of an incident response plan is to identify who has authority over an asset and who pays for it, too; they might not be the same unit

• Exercises have the potential to expose that information, at times with great relief

• Results should be included in plan review• Good justification for exercise


What Capabilities Succeed or Fail?

• Text paging has failed, but not noticed because the monitoring system pages the operators to report problems

• How many of you provision your support teams with toll-free numbers?

• How many of you know that toll-free dialing won’t be available in a disaster?

• Or that it can’t be dialed fromoutside the region (overseas)?


What are the Unforeseen Obstacles?

• Another obvious reason for an exercise; many hope to find the “gotchas” before a real crisis occurs

• Unfortunately, it’s based totally on luck• TIP: review your toll-free number uses• TIP: make sure your teams really know

how to use PGP and have hadtheir keys signed & published


What Serendipity Awaits Us?

• Exercises are a good thing, and every one in which I have participated has produced valuable results with practical application

• It’s easy to forget about positive stuff when we worry so much about negative things

• One example: other teams rewrote my faux advisory and discovered aspects that hadn’t occurred to me earlier


What Estimates Can We Calculate?

• Cyber security is catching up with metrics• Still horribly lacking with incident response• Exercises can expose unforeseen costs as

well as unanticipated rewards• Both help to reinforce the value of CSIRTs

to management up to the board room level• Also helps to reveal intangibles like

sharing opportunities and potential future relationships


What We Fail To Learn

• We fail to bring in the existing experts• We fail to discover existing stakeholders,

groups, capabilities, relationships• We fail to assess authority & responsibility• We fail to appreciate the resources and

time involved in anticipated responses• We fail to imagine the threats• We fail to keep it secure


We Fail to Bring in the Experts, 1

• FS-ISAC tabletop considered a power failure at a telephone switching facility due to sabotaged diesel backup systems

• Organizers unaware of battery systems and alternative fueling systems

• Credibility was suspended and the participants were unmotivated

• Value of exercise questionable


We Fail to Bring in the Experts, 2

• Exercise planners spent considerable time on scenario involving railroad cars and the lack of real-time tracking ability; expected major fumbling by participants to resolve

• In reality, locomotives are needed to move train cars & their locations are well known!

• As before, credibility was suspended, etc.• Exercise value plummeted!


We Fail to Discover Current Players

• “FIRST” means many things to many folks▪ The “Federal Incident Response

Support Team” might not be who you think it is; insist on clarification

• Misunderstandings about FIRST influence incorrect conclusions favoring involvement▪ Information sharing▪ Web of Trust


We Fail to Assess Authority, 1

• ISACs are defined per CIP sector for information-sharing and analysis▪ IT-ISAC handles information technology▪ Telecom-ISAC handles telephony▪ Who handles the ISPs? Each ISAC

says the other has superior authority• And the ISPs “just want to be

left alone, thank you…”


We Fail to Assess Authority, 2

• The U.S. National Response Plan divides activities by defined functional areas

• Emergency Support Function #2 handles telecom and information technology, while ESF#7 supports office equipment

• When a server in a disaster agency’s remote field office starts attacking other systems, who will handle it?

• Answer: “No one, immediately”


We Fail to Anticipate Response Cost

• Most plans (and thus most exercises) are oriented toward physical events

• In cyber-space, most planning ignores the international angle (Cyber Storm is trying hard to get this right, and will succeed)

• For example, for an international attack I was instructed to notify the Department of State’s 24-hour Watch Desk...

• Guess how long that takes!


We Fail to Imagine Threats

• Following Hurricane Katrina, IT/Telecom restoration initially followed rules oriented toward public safety, not toward critical infrastructure protection issues

• A major bank couldn’t get essential parts for back-office transaction processing

• “Instant cash” was unusable because bank was completely unreachable


We Fail To Keep It Secure

• Multi-site exercise connected to the Internet reduces cost but poses risks

• Collected diverse set of security experts connect to web pages for net simulation

• Traffic is not SSL-enabled nor tunneled• Links to “bad sites” were genuine and

HTTP referrers had not been disabled!• To their credit, Cyber Storm

staff fixed that within hours


Purpose of Exercises, Redux

• Pre-identify essential groups which will provide better coverage of stakeholders

• Improve rewards for active participation• Eliminate the “Yet Another Group” problem

(include FIRST; spell it out if necessary)• Constrain novelty-for-novelty’s-sake-alone• Identify and preserve your

group’s corporate knowledge


Future Improvements

• Invite the experts; inform them, trust them• Research and approach existing groups;

don’t start your own until and unless you are certain a collaboration will not succeed

• Test your equipment and methods as realistically and thoroughly as possible

• Assume issues are global• GET INVOLVED!


What Else?

• Contact information:James N. Duncan, [email protected]@gmail.com+1 919 334 4318 (office)+1 919 608 0748 (mobile)http://www.LinkedIn.com/in/JimDuncan/

• Questions and Answers?

duncan - what we learn from cyber exercises or not

Documents