Download - Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster
Simmons College April 8, 2006
Not the Unthinkable, But What We Didn’t Think Of
Preparing For and Recovering From DisasterCharles Patch
Director of SystemsThe Historic New Orleans Collection
Simmons College April 8, 2006
Hurricane Katrina was the most destructive natural disaster in U.S. history. The overall destruction wrought byHurricane Katrina, which was both a large and powerful hurricane as well as a catastrophic flood, vastly exceededthat of any other major disaster, such as the Chicago Fire of 1871, the San Francisco Earthquake and Fire of 1906,and Hurricane Andrew in 1992.
– The Federal Response to Hurricane Katrina, Lessons Learned. Department of Homeland Security, February 2006
Disaster in the Real WorldThe Official Word
Simmons College April 8, 2006
Living in a post disaster world
Simmons College April 8, 2006
Living in a post disaster world
Simmons College April 8, 2006
Living in a post disaster world
City population < half of pre-Katrina levels > 60% of city uninhabited 20 public schools in operation
15% of student body 5 of 14 branches of public library open Most museums and archives not open until late February
Simmons College April 8, 2006
Living in a Post Disaster WorldCondition of Public Libraries in LA Post-Katrina
Simmons College April 8, 2006
Disaster in the Abstract
Q: It’s Armageddon. You have 30 minutes to leave. You are carrying provisions for 3 days, your family and your pets. What is the one last thing you will take? From home? From Work? How long will it take you to find this thing?
Simmons College April 8, 2006
Disaster in the Abstract
Increases in scope and intensity increase impact of a disaster. Impact can vary in regard to timing:
As intensity increases, the greater the need for fast recovery measures The recovery of materials may be time-dependent Severely damaged physical plants need immediate stabilization
As scope increases, recovery becomes more difficult to execute Common services and infrastructure may be unavailable
Scope vs. Intensity can affect how you recover Materials Human Resources Property
Both scope and intensity hamper the execution of business processes A widespread disaster can prevent access to facilities as much as
an intense but localized disaster
Simmons College April 8, 2006
Responding to Disaster in the Abstract
Disaster Prep vs. Disaster Recovery vs. Business Recovery Disaster preparation = protection from uncontrollable
events: Protecting assets:
Materials Human Resources Property
Disaster Recovery is about the restoration of damaged assets and the recovery of lost resources
Disaster Preparation + Disaster Recovery Plan = Disaster Plan Disaster plans develop “Controls”:
Disaster Prep: install sprinkler system Disaster Recovery: stock conservation supplies
Simmons College April 8, 2006
Responding to Disaster in the Abstract
Disaster Prep vs. Disaster Recovery vs. Business Recovery Disaster preparation = protection from uncontrollable
events: Protecting assets: the “tiki hut” at the Museum of Natural History
in NYC – to protect network gear from an aging sprinkler system
Simmons College April 8, 2006
Responding to Disaster in the Abstract
Business Continuity is about the survival of basic business functions If you can’t function, you can’t recover
The larger/greater in scope/more intense the disaster, the more important business continuity becomes
Business Continuity is about managing the consequences of an adverse “risk event” in terms of basic business functions Specifically, business functions that have been interrupted
Simmons College April 8, 2006
Responding to Disaster in the Abstract
TERMS: Business Activities Business Continuity Plan Business Impact Analysis (BIA) Business Interruption Event Business Process Business Resource Business Resumption Plan Control Key Business Process Maximum Acceptable Outage (MAO) Outage Procedures Resources Risk Event Risk Management Plan Service Area Contingency Plan
Simmons College April 8, 2006
Responding to Disaster in the Abstract
Business Continuity Management (BCM) is part of a larger Risk Management Plan Disaster Recovery is a subset of BCM
Risk Management considers both negative and positive risk
The Prime Directive: Plan for the best, prepare for the worst Proactive design and implementation of
controls to prevent risks from occurring Reactive design of controls to mitigate effects
of adverse events that occur.
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Overview of Risk Management Process
Establish Context Determine key business objectives, processes and resources
Identify and Assess Risks
Identify, analyze, rate and prioritize risksEvaluate design of existing controls.Redesign controls if necessary
Implement Controls Establish plan. Implement controls
Monitor and Review Review operation and continuing suitability of controls
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Organizational Context
Organizational Objectives
Output Group Output GroupOutput Group
key business process
key business process
key business process
key business process
key business process
Business Support Process
Business Support Process
Business Support Process
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Organizational Context
Collect, Interpret, Display Cultural Materials for General
Public
AdministrationCollections
Management
Information Systems
Payroll
Facilities management
Accounts Payable
Acquisitions
Data storage
File hours with Paychecks Inc.
Create loan records in Collection Management
System
Create semi-weekly backups of all data
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Risk Management Process
No
Yes
No
Yes
Determine Possible Risk
Events using risk framework
Determine likelihood and consequence
without control in place
Determine risk level and
compare with acceptable risk
Acceptable?
Evaluate design of existing
controls
Determine likelihood and consequence with control in
place
Determine risk level and
compare with acceptable risk
Acceptable?
Redesign controls
Record in Risk Register
Identify
Analyze
Evaluate
Control
Document
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Risk Identification
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Analysis & Evaluation
Is a risk minor (acceptable) or major (unacceptable)? What is “acceptable”?
What are the consequences in terms of resources? Do they have a detrimental impact on
Staff Facilities Collections Telecommunications Information Systems
Rank the consequences on the degree of impact on these resources What is it worth to you?
A consequence can be minor if its fullest possible impact will not be detrimental to institutional resources.
Last question: will a consequence have an impact on normal business operations? Events which have detrimental impacts on resources are likely to
impact business operations <- remember this!
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Controls
Preventative Controls Stop the risk from occurring in the first place
E.g.: require passwords to access the computer system Ban flammable solvents from processing areas
Corrective Controls Minimize the consequence of a risk event once it has occurred
Sprinkler system Backup tapes for computer system
Evaluate current control mechanisms If the consequences remain unacceptable, redesign them
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Monitor and Review
Incorporate risk management into normal business operations Strategic planning Budget process Facilities Maintenance
Make risk management part of operational management Supervisors and key workers participate in planning and analysis Sign-offs by managers on risk controls
Practice procedures Structured walk-through Review for validity Confirmation of supplies needed
Simmons College April 8, 2006
Responding to Disaster in the Abstract: The BCM
A Business Continuity Plan is a Corrective Control An interruption in a Business Process is an adverse consequence
of a risk event A complete interruption of this sort is an Outage.
How long can an Outage be tolerated before the viability of the organization is threatened? This duration is the Maximum Acceptable Outage (MAO)
Risk Management
Conrols
Preventative
Corrective
Disaster Recovery
Business Continuity
Simmons College April 8, 2006
Responding to Disaster in the Abstract: The BCM
Business Continuity in 6 arduous steps1. Initiate the project:
Document Hypotheses, Scope and boundaries Establish a management committee Establish a budget and timetable
2. Identify Key Processes: Key institutional objectives Key “outputs” or end results of any process critical to the
institution Understand the key activities, resources and their
interdependencies Rank the importance and create an activity and resources
“schedule”3. Conduct a Business Impact Analysis:
Identify key personnel Conduct interviews Document Concerns Determine MAO
Simmons College April 8, 2006
Responding to Disaster in the Abstract: The BCM
4. Design Continuity Controls Review existing controls Identify and evaluate options and alternatives Select the alternative activities and resources
5. Implement the controls Establish recovery teams Document “Service Area” action steps Establish event escalation process Obtain and consolidate contact and inventory lists Document recovery management process
6. Test and Maintain the plan Paper test Manual Verification Supply validation Supply, Service and equipment availability Structured walkthrough Unannounced team assembly
Simmons College April 8, 2006
Responding to Disaster in the Abstract: Who Does All This?
Roles: Disaster Recovery Coordinator Service Area Teams Vendors / Service Providers
Simmons College April 8, 2006
Responding to Disaster in the AbstractThe Inconvenience of the personal lives of personnel
“Normal” personnel issues Employees with dependents Employees with physical disabilities Extended absences of employees with critical knowledge
Disaster issues that exacerbate personnel problems People with dependents need more time to prepare for an
evacuation People with dependents cannot respond as quickly to emergency
calls as those without dependents People with dependents may not be able to return as early as
others E.g.: no school in New Orleans until January 2006 E.g.: inadequate medical care
Simmons College April 8, 2006
Responding to Disaster in the Real World:
Background: The Yearly Drill Street flooding is common Hurricane / Tropical Storm alerts are yearly events in New Orleans
Simmons College April 8, 2006
Responding to Disaster in the Real World: Cindy – Wednesday 7/6/05
Simmons College April 8, 2006
Responding to Disaster in the Real World: Dennis –Sunday 7/10/05
Simmons College April 8, 2006
Responding to Disaster in the Real World: Katrina – Monday 8/29/05
Simmons College April 8, 2006
Responding to Disaster in the Real World: Living With the Past
Simmons College April 8, 2006
Responding to Disaster in the Real World: Living With the Past
Simmons College April 8, 2006
Responding to Disaster in the Real World: Preventative Controls
Keep Buildings maintained and up to code Fire alarms, fire extinguishers checked on semi-yearly basis Routine roof inspections Plumbing and electrical services kept to code or better Lighting and signage maintained properly
If you have the choice, don’t put things where they could be harmed
Simmons College April 8, 2006
Responding to Disaster in the Real World: Preventative Controls
Follow best practices for data management and storage including offsite backups
Make duplicates or copies of important documentation and store them offsite E.g.: All accession records and documentation microfilmed
at the end of each year, including updates to active collections
Stored in two difference places,
one offsite
Simmons College April 8, 2006
Responding to Disaster in the Real World: Corrective Controls
File plans with local fire and police departments
Be sure that room and floor designations are the same in all documentation, including insurance, collections management systems and fire department plans.
Keep current contact lists for staff
Consider requiring third-person contact numbers
Keep current contact lists for vendors, lenders, staff and emergency services
Simmons College April 8, 2006
Responding to Disaster in the Real World: Corrective Controls
Simmons College April 8, 2006
Responding to Disaster in the Real World: What We Did Right
Routine Best Practices Paper Accession records microfilmed and
stored offsite Computer records backed up routinely
and stored in separate buildings on a scheduled basis.
Payroll and benefits services handled offsite
Simmons College April 8, 2006
Responding to Disaster in the Real World: What We Did Right
Simmons College April 8, 2006
What We Did Right: Disaster Preparation
Disaster preparedness planTHE HISTORIC NEW ORLEANS COLLECTIONDISASTER PREPAREDNESS PLANTABLE OF CONTENTSI. Mission Statement/Staff Assignments Page 4II. Emergency Prevention Strategies Page 8III. Communication in the Event of Page 10
an EmergencyIV. Visitor Safety
A. Fire Page 11B. Illness
Page 11C. Accidents Page 12D. Robbery Page 12E. Vandalism Page 12F. Bomb Threat Page 13
V. Emergency Resources Page 15Building Resources Page 16Collections Resources Page 17
VI. SuppliesPage 20
VII. Fire Prevention Page 21VIII. Hurricane Preparedness Page 22
Simmons College April 8, 2006
What We Did Right: Staff Preparation
Regular testing of alarm system and drills Training of docent staff in emergency procedures All areas kept stocked with emergency supplies
Simmons College April 8, 2006
What We Did Right: Staff Preparation
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Did Right 3: Disaster Preparation
Prioritization of materials
Simmons College April 8, 2006
Responding to Disaster in the Real World: What We Did Right
Storm monitored by Disaster Coordinator from first appearance early in the week when still a tropical storm
Emergency supplies checked, calls made to confirm that all staff has copy of Disaster manual
Preparation Meeting held Friday morning, August 26th in directors office Emergency message activated Staff put on alert Head count of who is available this day and the next
Initial clean-up of processing areas completed Storm monitored throughout day Saturday, escalation of probable hit, staff called in at 7:30 AM,
16 people respond.
Simmons College April 8, 2006
Responding to Disaster in the Real World What We Did Right: Set Up
Saturday: Fresh computer backup tapes produced Collection Management System data and
Membership/Development data compressed and sent via FTP to vendors
Computer system shut down and disconnected from power source
Major servers moved to second floor of buildings and covered with plastic sheeting
Phone system shut down and disconnected from power source Art work removed from exterior walls
Simmons College April 8, 2006
Responding to Disaster in the Real World What We Did Right: Set Up
Sunday: Remaining staff either evacuates city or takes protective
measures at their homes Hurricane raised to Category 5, then Category 4 status; Direct
hit procedures invoked. Five staff return to carry out remaining preparations Removal of all works from floors and surface areas Relocation of all art works on first to second floors
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Did Right: Set Up
Building Preparation
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Did Right: Set Up
Building Preparation In-place precautions
Simmons College April 8, 2006
Responding to Disaster in the Real World: August 28, Sunday: The Evacuation
Mayor declares mandatory evacuation on Sunday morning
Extremely difficult travel conditions prevail 8 to 12 hours to reach Baton Rouge (80
mile) Director contacts all department heads by
phone to establish whereabouts Director and several other staff members
stay to ride out storm
Simmons College April 8, 2006
Responding to Disaster in the Real World: 8/29 – 9/3 : Katrina “misses” New Orleans
Hurricane tracks to the East of the city Power and phone service throughout
Southern part of state lost during most of Monday
Monday night, most believe New Orleans has “dodged the bullet” again and plan return to the city on Tuesday morning.
Tuesday morning television reports 80% of city is under water
Phone service dysfunctional, cable and Internet access extremely rare
By Wednesday there are 100,000 extra people in Baton Rouge
Google Group for staff established on Thursday night
Initial reports suggest it will be months before the city can be inhabited again
THNOC arranges to participate in a state police convoy. Due to dangers in the city, the convoy is cancelled
Simmons College April 8, 2006
Responding to Disaster in the Real World What we did right: The Evacuation
We learned to work the damaged phone system Lesson: cell phones worked via text message in the New Orleans area and
with voice when calling outside the New Orleans area code. First Contact: Tuesday night, August 30
Staff members in the area organize rescue of valuable materials. Initially planned for September 2, delayed because of conditions until
September 8 “High Priority” materials were clearly marked and were stored in easily
accessed locations Offsite location for materials established upstate Movers retained on special contract called in
Simmons College April 8, 2006
Responding to Disaster in the Real World What we did right: The Evacuation
Google Group created 9/1 Staff assured of continued employment, salary and benefits
This is the first message posted to the list that is not a response to the “Where are you?” query.
Simmons College April 8, 2006
Responding to Disaster in the Real World What we did right: The Evacuation
Staff web page created 9/5
Simmons College April 8, 2006
Responding to Disaster in the Real World What we did right: The Evacuation
Personnel Database created 9/13
Simmons College April 8, 2006
Simmons College April 8, 2006
Simmons College April 8, 2006
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat Went Right: The Evacuation
Staff stayed of their own volition Our vendor becomes our go-between
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat Went Right: The Evacuation
Staff members found the Google Group Our web master took the site with him on his laptop Our network manager was close enough to join the
convoy Our vendor anticipated our need for a personnel database The French Quarter was spared and sufficient staff were
on hand to open the facility on October 3.
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Could Have Done Better
No designated essential personnel Compounded by timing issues
Inadequate communications procedures Dependence on phone system
for communications No working email addresses No common contact point for
reaching individual staff members
No alternative admin site And no policy for defining how
to recognize a workable alternative
No back-up network site Inadequate conservation supplies on
site Most suppliers for recovery
materials listed in handbook were local
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Could Have Done Better
Some things we didn’t think (that might have been caught in a risk analysis) Book Trucks The Real Truck:
Who was going to drive this?
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Could Have Done Better
Some things we didn’t think (that might have been caught in a risk analysis) The giant image files that weren’t backed up
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat We Could Have Done Better
Some things we didn’t think (that might have been caught in a risk analysis) Network “single point of failure” (actually 4 potential failure points) Magnetic Locks (no electricity, no security)
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat Didn’t Go Well?
A failure to communicate Administration chose to stay close to the physical institution rather than
communication hubs. Most were in rural areas with poor or no Internet access Nearly all communication from the administration was by cell phone,
which worked only sporadically Administration did not take advantage of communications channels
Tended to wait until staff contacted them rather than contacting staff Rarely posted information on the staff web page or the Personnel DB
bulletin board
Time lost to making ad hoc arrangements No pre-arranged alternative “home” for rescued collections No pre-arrangements for care of “essential staff” before or after
disaster No plan for setting up network in alternative location while staff was
disbursed (as opposed to securing backups)
Simmons College April 8, 2006
Responding to Disaster in the Real WorldWhat Didn’t Go Well?
Simmons College April 8, 2006
Responding to Disaster in the FutureLessons Learned?
Most forms of communication will fail No single source will provide adequate information concerning real
events Individuals or small groups working independently were most effective Data redundancy is highly desirable Professionals involved in activities unrelated to what authorities regard
as the central economic interests of the city and region will be marginalized
Simmons College April 8, 2006
Responding to Disaster in the FutureLessons Learned?
Communications 101 Mechanics:
Create multiple forms of asynchronous communication Out-of-Region phone bulletin board Web discussion groups/web pages/staff directories “Real” bulletin board on site for leaving paper messages to
other staff Develop policies for choosing administrative sites that
provide for access to the disaster area AND good communications
Simmons College April 8, 2006
Responding to Disaster in the FutureLessons Learned?
Communications 201 Semantics:
Create a protocol for communicating with staff Procedures for posting staff supplied information
Scheduled administrative updates and news “Test the channel”
No news is bad news
Simmons College April 8, 2006
Responding to Disaster in the Future Information
“We heard stories about helicopters being shot. But you’ve been in helicopters, and you know how noisy they are. The only way you know you have been shot at is if there’s a bullet hole,” Landreneau said. “There were no shots fired at our helicopters.”
-Maj. Gen. Benny Landreneau, Louisiana National Guard commander , quoted in “What We Signed Up For” by John Hill, Louisiana Life Magazine, March 2006
Information: Who To Believe?
Simmons College April 8, 2006
Responding to Disaster in the Future Information
The New York Times has confirmed that one person was murdered at the Convention Center and one at the Superdome, and the Times-Picayune has confirmed that a National Guardsman was attacked by an assailant wielding a metal rod in the darkened Dome. The coroner’s early report implies that the murder rate among those stranded in Katrina’s aftermath was at least five times New Orleans’s normal murder rate. This real, not imagined, violence prevented New Orleans from getting the level of volunteer and professional help it needed after Katrina. - “Who’s Killing New Orleans?” Nicole Gelinas, City Journal, Autumn 2005, http://www.city-journal.org/index.html
Information: Who To Believe?
Simmons College April 8, 2006
Responding to Disaster in the Future Information
If the images were to be reduced to a sentence in the minds ofUptown New Orleans, that sentence would be: Crazy black people withautomatic weapons are out hunting white people, and there's no bag limit!"The perspective you are getting from me," one of Fort Huger's footsoldiers said, as he walked around the living room with an M-16, "is theperspective of the guy who is getting disinformation and reactingaccordingly.“
-Michael LewisWading Toward HomeNew York Times Magazine, Oct 9, 2005
Information: Who To Believe?
Simmons College April 8, 2006
Responding to Disaster in the Future Information
The media’s willingness to report thinly attributed rumors may . . . have contributed to a cultural wreckage that will not clean up easily. . . . Victims, officials and reporters all took one of the most horrific events in American history and made it worse than it actually was,” Lecture by New York Times media reporter David Carr, September 2005
“Four weeks after the storm, few of the widely reported atrocities have been backed with evidence,” - Times-Picayune September 26, 2005
Information: Who To Believe?
Simmons College April 8, 2006
Responding to Disaster in the Future Information
Weigh all information carefully and whenever possible, rely on trusted sources in the area
Information: Who To Believe?
Simmons College April 8, 2006
Responding to Disaster in the FutureInformation: Local Discussion Lists
Simmons College April 8, 2006
Responding to Disaster in the FutureInformation: Local Discussion Lists
Simmons College April 8, 2006
Responding to Disaster in the Future Facilitate Independent Agency
Provide materials for short term survival as well as collections care.
Leverage the actions of individuals by providing some method of coordinating among disparate players
“Response to Katrina is less dependent on traditional disaster plans than on improvised actions as conditions permit.”-Report of Hurricane Katrina Damage Assessment:Debra Hess Norris (Heritage Preservation)Richard Pearce-Moses (Society of American Archivists)David Carmichael (Council of State Archivists)21 September 2005
Simmons College April 8, 2006
Responding to Disaster in the Future Have Have More More Than One One
Off-site storage of data Meta-data tagging of individual digital
assets Cooperative arrangements for
network operations with sister institutions
Cooperative arrangements for physical storage with sister institutions
Simmons College April 8, 2006
Responding to Disaster in the Future Have Have More More Than One One
Just a few miles west of Pass Christian, the Hancock County Historical Society in Bay St. Louis fared much better with very little water damage and a vault that held, protecting thousands of documents, including family diaries and thousands of local photographs.
Charles Harry Gray, the executive director, was prepared in case disaster struck. Over the years he had been making copies of all of the group's most treasured documents, including 30,000 pictures. Not one single photograph or record was lost.
They are the pieces of Bay St. Louis' 306-year history that made the town of 8,230 what it is today, he said. Many of the copies were on computer disks and hard drives, others were sent to the University of Southern Mississippi, two hours north in Hattiesburg.
"It is imperative that you have copies in other locations because you never know what's going to happen, what the next catastrophe is going to be, and there certainly will be one," Gray said.
Simmons College April 8, 2006
Responding to Disaster in the Future Disenfranchisement
Though archivists began asking for re-entry into the city on August 31…(all) requests for attention to historic paper records were denied…The Department of Culture, Recreation and Tourism failed to include archivists on planned reconnaissance trips into the city or to include anyone from the archival community in its planning meetings
Susan TuckerCurator of Archives at the Newcomb College Center for Research on Women Email, 10/14/2005
Simmons College April 8, 2006
Responding to Disaster in the Future Disenfranchisement
Simmons College April 8, 2006
If this ever happens again…
All staff should have a web-based email address . Publish and update as part of the recovery documentation
All staff should provide a contact phone number outside of the region We should establish a telephone bulletin board outside of the institution We should consider a subscription to a satellite phone service All department heads should have lap-top computers Online forum should be set up permanently with a single instructional
message At least one institutional email address that is hosted on an off-site
server Staff-only emergency web page should be set up permanently Policy for alternative administrative site
Not necessarily choose an actual location but develop criteria for choosing one
Develop cooperative arrangements with other institutions for storing collections
Develop cooperative arrangements with other institutions to host our computing equipment and run our network
Contract for archiving of digital assets that are difficult to transmit or store easily to tape
Simmons College April 8, 2006
Web Sites of Interest
Business Continuity Management NEDRIX - New England Disaster Recovery Information X-cha
nge http://www.nedrix.com/
Risk Management Standards http://www.incom.com.au/enterprise-risk-management-standards.htm
Better Practice Guide - Business Continuity Management - Keeping the wheels in motion http://www.anao.gov.au/WebSite.nsf/Publications/
4A256AE90015F69B4A2568EE0010062B National Archives of Australia - Business continuity plannin
g for digital records http://www.naa.gov.au/recordkeeping/er/guidelines/9-continuity.html
Business Continuity Planning & Disaster Recovery Planning World http://www.disasterrecoveryworld.com/