9/11 - respond and recover: a case study joy heath-porter sidley austin brown & wood llp lawnet...
TRANSCRIPT
9/11 - Respond andRecover: A Case Study
Joy Heath-PorterSidley Austin Brown & Wood LLP
LawNet 2002August 21, 2002
Who are we?
• Large, multi-office, multi-national firm
• 14 offices; six countries
• 1,500 lawyers; 3,000 total personnel
• Two offices in New York before September 11; 600 person office in the North Tower of the World Trade Center
Status on September 10
• Involved in one of the largest law firm mergers ever
• Two networks with different hubs
• Two different e-mail and document management systems
• Two different telephone systems
Status on September 10(cont’d.)
• Same financial system, but on different versions with different database engine
• Still in the process of consolidating technology
• Had just moved equipment to WTC previous weekend
Brief Chronology
• 7:50 a.m. Tuesday - how we found out; first steps
• Mid-morning Tuesday - disaster recovery steering group assembled; began to map out recovery strategy
• Noon Tuesday - offsite storage provider contacted; arrangements made to drive tapes to Chicago
Brief Chronology (cont’d.)
• Throughout Tuesday - made sure phones were manned, so that staff calling in would have a “live” voice; determined status of our people; identified and arranged for recovery resources; Chicago and New York offices “closed,” but many staff members were working
• C.O.B. Tuesday - system recovery priorities established; servers identified for recovery
Brief Chronology (cont’d.)
• Tuesday night - 800 dial-in information numbers established and published; information made available via the Firm’s Web site; temporary space identified
• Wednesday morning - all offices contacted for spare equipment; arrangements made for rental equipment; PC vendor contacted for purchase of equipment; cabling began
Brief Chronology (cont’d.)
• Wednesday noon - Tapes arrived; restoration process began
• Wednesday p.m. - voice mail accounts on alternate voice system began to be created for lawyers
• Wednesday night- new e-mail server built; account creation in process; Internet mail redirected (brownwoodlaw.com)
Brief Chronology (cont’d.)
• Thursday a.m. - all lawyers had e-mail accounts and mail was flowing; equipment was shipped out from other Firm offices
• Thursday p.m. - Internet café established in alternate NY sight; image build began for rental equipment
• Thursday night - secure extranet site established for technology information
Brief Chronology (cont’d.)
• Friday a.m. - document management was available to NY users
• Friday noon - equipment began arriving from other offices and from vendor
• Friday p.m. - financial systems restoration was complete and consolidation began; image for rentals and new PCs finalized
Brief Chronology (cont’d.)
• Saturday a.m. - Rental equipment began arriving; imaging began; teams from Chicago began driving to New York transporting equipment not available in New York (cell phones, card keys, etc.)
• Throughout weekend - Phones, network equipment, printers, desks, PCs rolled out in temporary space
Six Days Later
• Monday 10:00 a.m. - all New York personnel assembled in a mid-town hotel for a briefing
• Monday noon - all New York personnel reported for work in temporary space
Today
• All servers returned to New York
• All technology migrated to firm standards
• New York office moved into new, permanent home over July 4 weekend
Key Aspects of Disaster Recovery Approach
• “Maniacal” approach to tape backups - daily offsite storage of tapes; offsite location outside metropolitan area
• Focus on running our daily operation to perfection
• Standardized hardware (network, PCs, phones) with standard images
Key Aspects of Disaster Recovery Approach (cont’d.)
• Emphasis on documentation and standard operating procedures
• Commitment to serving our lawyers the way our lawyers serve the Firm's clients
Lessons Learned/Affirmed
• You cannot plan for a plane hitting your building - accept the fact that you will have to improvise to some degree in virtually any disaster
• You cannot have too many friends - clients, vendors, co-counsel, even “competitors”
• Practice, practice, practice - staff need to be able to perform basic disaster recovery tasks "in their sleep"
Lessons Learned/Affirmed(cont’d.)
• If you have multiple offices, leverage them
• The fact of back-ups is not enough - they must be quality-checked, verified, and viable
• Efficiencies of centralization must be weighed against risks from a disaster recovery perspective
Lessons Learned/Affirmed(cont’d.)
• Recruit for character and creativity, as well as technical expertise
• Understanding the relative priority of your critical applications cannot be overstated - in a disaster situation, restoring contact information may be more important than restoring documents
Lessons Learned/Affirmed(cont’d.)
• Web access to your systems is not a luxury; it is a necessity
• Stress with principals the importance of using firm-standard repositories - if documents are in three different places, restoring access becomes much more difficult
Lessons Learned/Affirmed(cont’d.)
• Think about disaster recovery when making a buy/build decision
• Smart, dedicated people ultimately make it happen
How does theStory End?
• We were never “down”
• Clients were impressed
• Our most important assets - our smart and dedicated people - were not lost
• The loss of people is the only disaster truly to fear - if you are prepared, everything else can be recovered or replaced