analysts international business continuity planning v1.01 10-2002

Analysts International

Business Continuity Planning

V1.01 10-2002

2

Introductions• Mark Lachniet from Analysts International,

Sequoia Services Group• Senior Security Engineer and Security

Services technical lead• Former I.S. director for Holt Public Schools• Certified Information Systems Security

Professional (CISSP)• Microsoft MCSE, Novell Master CNE, Linux

LPI Certified LPIC-1, Check Point Certified CCSE, etc.

3

Analysts International At a Glance

• More than 30 years of technology experience• Staffing and Sequoia Services• Employees: 4,800• Nearly 40 locations• More than $550 million in annual revenue• More than 1,000 clients• Focus on organization and planning• Strong partnerships with industry leaders –

Cisco, Microsoft, Citrix, XIOtech, etc.

4

The Need for Business Continuity Planning

• Disasters happen – its probably a matter of *when* and not *if*

• Of those organizations that have a disaster that interrupts them for more than a few days, many of them are out of business within 5 years

• Chicago’s flood in 1992, caused by an errant contractor with a drill, filled basements and telecom tunnels with water for days

• When eBay was down for 22 hours in 1999, the estimated loss of income was between $3Million - $5Million

5

September 11th, 2001• Unfortunately, 9/11 has brought many companies to

assess their own internal recovery plans

• 9/11 also expanded our understanding about what types of disasters were realistic – it is no longer just tornados and earthquakes

• The World Trade Center - September 11th, 2001 took out not only businesses, but the telephone and Internet routing hubs in the basement affecting much of the city

• It will cost an estimated $3.2 Billion to replace the IT systems lost in the World Trace Center

• More than half of that cost will be for software and labor to install the systems

6

Untangling the Mess• There is not always consensus on what

terminology to use, and what approach to take. This can make it difficult to sort out. Here is our take on it:

• Business Continuity Planning (BCP):– Focuses on the entire business: operations,

emergency procedures, insurance, facilities, etc.

• Disaster Recovery Planning (DRP):– Focuses on the technology: procedures to

minimize risk beforehand and recovery procedures to get things up and running again after a disaster

7

The #1 Priority

Protect PEOPLE!

8

Before You Start• Obtain concrete management support • Research Research Research• Plan Plan Plan• Leave yourself a lot of time• Make lots of coffee – schedule mornings • Be aware that BCP work must go on forever

and cannot ever truly be finished• Break your Business Continuity Planning into

multiple phases• Finish a phase and then evaluate it before

moving on

9

Analysts International’sPhased Approach

Phase 1Risk Assessment

Phase 2Plan

Development

Phase 3Plan Testing

Phase 4Plan

Maintenance

10

Phase 1 – Risk Assessment

DefineScope

ProjectInitiation

Phase 1 - Risk Assessment

RiskAssessment

ApplicationsIdentifyTeam

DevelopSchedule

BusinessImpact

Analysis

Draft FormalRA Report

DocumentFindings

Findings

Next Steps

Org. PracticesPhysical Sec.

Fire PreventionPower and UPS

Data BackupSystem SecurityNetwork Design

Recommendations

CollectData

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

11

Phase 1 Overview• Analyze the existing environment • Make recommendations for both technological

improvements (such as hardware and software) and Business Continuity process and procedure development.

• Uses tools such as a Risk Assessment and Business Impact Analysis to analyze and document the existing business and technical environments

• A key outcome of this phase is the identification of: Critical business processes The applications that support these processes The tangible assets necessary for these applications to function.

• These identifications are then documented, analyzed for interdependencies, and used for establishing recovery priorities.

12

Phase 1 – Project Initiation• Define the scope of your BCP project:

Which locations are you going to cover? Are you going to cover a lot of non-technical systems and

plans? If so, do you have the requisite skillset to do this well? Do you have the legal authority? (think about the importance of a facilities evacuation plan)

How many systems are you going to try to plan for? Do you need to be “complete” or are you going to look at just the most critical systems?

Do you have any existing inventory databases or lists that you can use to identify your critical systems?

Setting scope defines the amount of time and effort you will put forth in future steps

Setting scope defines the responsibilities of vendors who are working with you on a BCP project so that they can try to cost-estimate the project

DefineScope

ProjectInitiation

IdentifyTeam

DevelopSchedule

CollectData

13

Phase 1 – Project Initiation• Collect data about the environment:

Existing BCP / DRP documents (y2k?)Existing inventoriesNetwork mapsServer listsIP Addressing plansOrganizational Hierarchy ChartsNetwork mapsService and maintenance contracts with support

vendorsInsurance coverageServer and system build instructions

DefineScope

ProjectInitiation

IdentifyTeam

DevelopSchedule

CollectData

14

Phase 1 – Project Initiation• Identify the BCP planning team:

Management representative (preferably with budgetary authority)

Knowledgeable functional area experts (department heads, managers, head secretaries, people who really know what goes on)

Facilities representatives (physical plant folks who know about systems such as telephone, alarm, fire detection, generator backup)

A “BCP Coordinator” or champion for the organization who will help coordinate schedules, distribute information, pester

Technical representatives:• Infrastructure (networking – firewalls and routers)• Server support (server OS and hardware)• Desktop / application support• Application development• Database managers

DefineScope

ProjectInitiation

IdentifyTeam

DevelopSchedule

CollectData

15

Phase 1 – Project Initiation• Develop your schedule

Plan at least a few weeks outTry to meet at least once a week, if not moreFocus your attendee list on the subject at hand,

don’t invite everyone to every meeting, especially high level administrators

Have a comfortable meeting space with a whiteboard and/or sketch pads, etc.

Don’t let the daily “emergencies” take a higher priority (hence one of the reasons for management support)

DefineScope

ProjectInitiation

IdentifyTeam

DevelopSchedule

CollectData

16

Phase 1 – Risk Assessment (Vulnerability Assessment)• An objective look at the posture of

disaster recovery and prevention systems• Create a checklist of objective measures

of compliance with best practices and industry standards

• Frequently IT and Facilities intensive• A comprehensive R.A. tool is a big benefit

of getting professional help• Must address a variety of topics from

physical security to generator backup systems

RiskAssessment




17

Phase 1 – Risk Assessment Topics

Information Systems standardsServer resiliency (RAID, hardware redundancy)Error monitoring and proactive detectionUPS and generator backup availability and

configurationVirus Detection systemsData backup procedures and off-site storageNetwork resiliency and single points of failureVoice infrastructure resiliencyFire detection and suppressionEnvironmental monitoring and control systems

RiskAssessment




18

Phase 1 – Business Impact Analysis

• Identify Business Processes (the core business of the organization – what do you do?)Accounts Payables and ReceivablesHuman ResourcesMarketingResearch and DevelopmentMaintain public web siteInternal training initiativesEtc.

• Brainstorm a list of everything you think you do in all depts.

Applications

BusinessImpact

Analysis

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

19


• For each high-level process, what are the tasks necessary to do it? Example: Hire Employees

Make decision to hireDraft job postingPost job postingAccept job applicationsStore resumesPerform interviewsCall referencesHire employees

Applications

BusinessImpact

Analysis

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

20


• For each task, determine the inputs and outputs of the task.

• Inputs: information or resources needed to perform the task

• Outputs: information or resources created or modified by the task Example: Draft job posting

Input: Employee Head Count (FTE’s)Input: Open position informationInput: Upcoming projects and needsOutput: Job description document

Applications

BusinessImpact

Analysis

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

21


• Identify all of the applications that are required for the task’s inputs and outputs Example: Draft job posting Application: SquidTech Human Resources DB Application: Contract database Application: Sales forecast DB Application: Microsoft Word

Applications

BusinessImpact

Analysis

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

22


• For each application needed to support a task and business process, what assets are required to support it? Example: Human Resources DB

Asset: Fileserver #1 (Compaq DL-320, 512, 9gig) Asset: Windows 2000 Advanced Server Asset: SQL Server version 7.0 Asset: XIOtech Storage Area Network Asset: SquidTech Human Resources server S/W Asset: SquidTech Human Resources client S/W

• This is used later to assign weight to assets such that assets that support a lot of applications have a high priority for protection and recovery procedures

Applications

BusinessImpact

Analysis

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

23


• Establish recovery requirements for each process• Some business processes will have tasks that are

more important than others• For example, you can take your time making a

decision to hire, but you usually need to check references and hire someone by a posted deadline

• Quantify how long you can afford to be without the task using some standardized scale: – 5: <4hrs– 4: 1 day – 3: 3 days – 2: 1 week – 1: 1 month

• This lets you establish weights and priorities for your systems

Applications

BusinessImpact

Analysis

RecoveryObjectives

BusinessProcesses

Assets

TasksInputs

Outputs

24

Phase 1 – Other Steps

• Identify resource requirements

• Identify legal obligations and contracts

• Identify insurance coverage

• Analyze vendor relationships

• Analyze support agreements

25

Phase 1 – Document Findings• Pull it all together and document it• Create recommendations for products that

should be considered• Create recommendations for the

development of preventative and recovery plans for critical assets and applications

• Create a roadmap for continued BCP planning

• Present information and documentation to management

• Make decisions on how to proceed – do you want to concurrently purchase any products

Draft FormalRA Report

DocumentFindings

Findings

Next Steps

Recommendations

26

Phase 2 – Plan Development

Detailed Planning

Draft FormalPlan

Document

Phase 2 - Plan Development

Define Scope

ProjectInitiation

Identify Team

DevelopSchedule

Review Data

Organizational Control

Data Backup/Safeguards

Emergency Operations

Damage Assessment

Physical Site Security

Facilities Recovery

Network Recovery

Info Systems Recovery

Transition to Normal

Next Steps

DocumentProcesses

27

Phase 2 - Overview• Detailed recovery procedures are developed for those

critical assets and applications identified in the Risk Assessment phase.

• These procedures include technological recovery procedures, such as building servers and restoring data

• Also include operational procedures such as those needed to evacuate a facility or manage contact with public service agencies and the media.

• This phase works out the details – for example the logistics of the emergency operations center, roles and responsibilities, and the primary and backup individuals for these roles.

28

Phase 2 – Project Initiation• This should look familiar….• Which plans are you going to write? • Which systems do you need to plan for?• Are you going to cover operational

planning? Do you have the expertise?• Have you thoroughly digested the results

of phase 1?• Who needs to be on the team? Can you

excuse some folks at this point? • Who is essential? (hint: knowledgable

functional area experts)

Define Scope

ProjectInitiation

Identify Team

DevelopSchedule

Review Data

29

Phase 2 – Detailed Planning• Preventative and recovery plans• Most technically detailed and requires

access to servers, documentation, media and licensing information (to name a few)

• Requires technical skills as well as project management skills

• Should utilize multiple individuals working in parallel to save time

• Will probably require access to legal and human resources as well as technical staff

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



30

Phase 2 – Organizational Control

• Designed to establish some of the roles and responsibilities associated with the Business Continuity Plan

• This is a sort of “primer” for Phase 4

• Identification of BCP responsibilities and teams

• Development and maintenance of notification lists and contact information

• Create systems for the maintenance of documentation and records

• Plan communication and training requirements – who needs to know and how will they learn?

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



31

Phase 2 – Data Backup and Safeguards

• Required hardware and software configurations & minimum standards

• For tape backup systems:Server Hardware and OS standardsTape drive / library configurationSoftware version standardsBackup job configuration standardsFrequency of backupType of backup (full, differential, incremental)Backup options (read-verify, CRC, etc.)Tape rotation and testing schedulesOff-site storage standards and procedures

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



32

Phase 2 – Data Backup and Safeguards

• For non-tape backup systems Disk mirroring and array configurations File mirroring and replication Storage Area Network usage and standards Hot-standby systems and facilities Batch copies, database exports

• Operational preventative measures Standards for the use of “server class” hardware UPS and generator backup power Server health monitoring Server physical security and protection Standards for the maintenance and safeguarding of data

on client and non-server systems

• Others as appropriate for environment

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



33

Phase 2 – Emergency Operations

• Document existing personnel safety considerations• Document existing evacuation plans • Declaration of emergency

Authorized individual Backup individual

• Initiate contacts When and how to start employee call trees When and how to make partner, supplier, and vendor contacts When and how to make contact with insurance companies When and how to make media contact When to contact public utilities (power, water, fire department,

law enforcement)

• Activation of teams Team roles and responsibilities Role primary, secondary and tertiary individuals

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



34

Phase 2 – Emergency Operations

• Emergency Operations Center Method and/or location to meet (may be virtual) Alternate location to meet Communication capabilities (phone, fax, Internet) Size of the facility Capabilities for food and lodging

• Critical resources to gather Off-site tapes and backups Software installation media and licenses Hardware Vendor-supplied equipment or contracted services Office supplies Cellular phones and pagers Emergency notification systems (weather radios, police

scanners, etc.)

• Communication management and logging procedures

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



35

Phase 2 – Damage Assessment• Definition of salvageable vs. non-

salvageable • Declare that items are salvageable or

non-salvageable • Computer hardware, software and

licenses • Network equipment • Books and paper records • Facilities (phone, HVAC, water, fire

prevention, alarm system)

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



36

Phase 2 – Physical Site Security

• Define roles and individuals responsible for physical monitoring of locations

• Distribution of keys, security codes as needed

• Physically monitor old and new work sites and emergency operations center

• Prevent inappropriate system activities (theft, system modification, etc.)

• Monitor old and new site environmental safety

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



37

Phase 2 – Facilities Recovery• Evaluation of primary location safety • Decision to recover at normal or

backup location • Power, UPS, and Generator Backup • Telephone installation, configuration,

and switchover • HVAC, water, fire prevention, alarm

systems • Physical security processes • Workspace considerations

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



38

Phase 2 – Network Recovery• Alternate site preparation• Network hub/switch installation and

configuration • Cable and cable management • WAN circuit backup procedures• WAN circuit recovery procedures• ISP recovery procedures (incl. DNS) • Workstation and server network

configuration and settings • Alternate communication procedures

(dialup, etc.)

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



39

Phase 2 – Information Systems Recovery

• The most complicated and detailed section the plan’s sections

• Hardware, software and license installation requirements

• Alternate site preparation (rack space, raised floors, etc.)

• Required steps to obtain new or alternate hardware

• Required steps to recover user workstations

• Required steps to recover critical servers • OS installation

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



40

Phase 2 – Information Systems Recovery

• Restoration of data• Printing systems• Insurance coverage • Third-party contracts • Support agreements with vendors • Customer-specific procedures • Prioritization of systems to recover• Written documentation of work performed• Additional organization-specific

procedures

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



41

Phase 2 – Transition to Normal Operations

• Declaration of the end of the emergency• Order and priority of moving computer

services to regular computing environment • Order and priority of moving facilities

services to regular computing environment • Decommissioning of the emergency

operations center • Compile information collected, logs, notes,

contracts and communications for future review

Detailed Planning




Damage Assessment


Facilities Recovery

Network Recovery



42

Phase 2 – Document Processes

• Combine all plans, procedures, forms, and documentation into a formal plan document

• Present the plan for review• Make modifications based on input• Formally accept the plan• Create paper, PDF and/or HTML

deliverables • Plan next steps

Draft FormalPlan

Document

Next Steps

DocumentProcesses

43

Phase 3 – Testing and Implementation

Designindividual

system testprocedures

Test PlanDesign

Phase 3 - Testing and Implementation

Define thepurpose,

team,resources

andapproach

TestingDefinition

Conduct thetests,

documentand analyzethe results

PerformTests

Refineprocedures

andprocessesbased onlessonslearned

UpdateRecoveryStrategies

Adopt theplan,

performtraining andawarenessinitiatives

FormallyImplementBCP Plan

44

Phase 3 - Overview

• The recovery procedures developed in the previous phase are tested

• By testing in a controlled manner, omissions, errors and lessons-learned are identified and used to further refine the Business Continuity Plan.

• The Business Continuity Plan is then updated and submitted for final acceptance.

45

Phase 3 - Testing Definition• What are the purpose and goals of the plan testing

in general?• Do you desire a “full interruption” test, a parallel

test, or some variation?• How many tests are required? • Will a hot-site or cold-site be used?• Will we be exercising the capabilities of vendors

and contracted services?• What resources will be required? Who will be

involved internally and externally?• This exercise be viewed as a type of training as

well – are all the right people involved?• Will operational plans be tested as well as

technological plans?

Define thepurpose,

team,resources

andapproach

TestingDefinition

46

Phase 3 - Test Plan Design• This is a critical part of the planning process• Create a detailed scenario for how each test will be

conducted including: The recovery procedures developed in Phase 2 Customer IT staff roles and responsibilities BCP consultants’ roles and responsibilities Testing timelines and sequence of events Expected outcomes Pass / fail criteria Back-out procedures in the event of a failed test

• Create a document for recording test results• Prepare and present the plan to the team that will

be involved in the testing• Coordinate with internal and external resources in

an organized fashion

Designindividual

system testprocedures

Test PlanDesign

47

Phase 3 - Perform the Tests• As per the previously created test

scenario, perform the test

• Keep detailed notes of how things went during the test:How long individual tasks tookAny steps that were omitted during planningAny dependencies that were not previously

identifiedRecovery priorities that should be changedThe actual outcomes of individual tests versus

what was expected

Conduct thetests,

documentand analyzethe results

PerformTests

48

Phase 3 - Update Recovery Strategies

• This step is done after all of the testing has been completed

• Debrief the BCP test team and discuss how things went

• Brainstorm on shortcomings, possible improvements, and additional changes to the plan

• Update the BCP plan with “lessons learned”

• Review the document and present the updated document for formal approval

Refineprocedures

andprocessesbased onlessonslearned

UpdateRecoveryStrategies

49

Phase 3 - Formally Implement the Plan

• Formal management adoption of the plan• Work on developing a training program for

technical staff: – Identify key staff to be trained – Identify subject areas to be included in training – Identify curriculum and procedures to be covered – Identify baseline technology skills necessary for Disaster

Recovery training – Identify the desired regularity of BCP training

• Work on developing a training program for non-technical staff:– Summaries of the content of the BCP plan– Telephone call lists– Media contact procedures– Building evacuation procedures

Adopt theplan,

performtraining andawarenessinitiatives

FormallyImplementBCP Plan

50

Phase 4 - Maintenance

Phase 4 -Maintenance

Regularlyreconvene theBCP planning

committee,compare the

written plan tothe current

environment,propose

changes asneeded to

update

OngoingMaintenance

As needed, updatethe plan using

established changecontrol systems.For significant

changes to the plan,proceed to Phase 1

for a RiskAssessment. Forsmaller updates,

proceed to Phase 2to update individual

components

Update PlanPlan for

Maintenance

IT Schedules

BCP Policies

Formal Roles

Job Descriptions

BudgetaryPlanning

Change ControlSystems

51

Phase 4 - Overview• Once the plan has been formally accepted, it

must be maintained in perpetuity• Business Continuity Plans can quickly

become outdated, making them essentially useless to the organization

• To minimize this risk and to put into place the necessary systems and expectations to support ongoing Business Continuity Planning efforts, a number of steps must be taken

• In particular, administrative policies, formal responsibilities and training initiatives must be developed and delivered.

52

Phase 4 - IT Department Schedules

• Create a formal schedule for the organization’s staff that formally revisits the Business Continuity Plan on a regular basis

• What is the regularity? (quarterly, yearly?)• What events will activate a mandatory re-

assessment? (new systems, big changes, new staff?)• How will it be coordinated with other regular

activities such as testing of UPS and generator backup systems, financial cycles, holidays and vacations, etc.

• What are the IT staff time commitments and time frames for the work?

• What are the management oversight duties pursuant to the ongoing work?

Plan forMaintenance

IT Schedules

BCP Policies

Formal Roles

Job Descriptions

BudgetaryPlanning


53

Phase 4 - Create BCP Policies

• Formal, high-level support of BCP initiatives is critical

• This requires formal statements of support by management and the board

• BCP policies should be created that include aspects such as: Recognition of the criticality of Business Continuity

Planning Statement of support for Business Continuity Planning

within the organization Formal expectation that Business Continuity Plans will

be regularly updated and tested

Plan forMaintenance

IT Schedules

BCP Policies

Formal Roles

Job Descriptions

BudgetaryPlanning


54

Phase 4 - Formal BCP Roles• There must be formal responsibility for BCP work

based on job roles• Individuals may fill these roles, but the

responsibilities should not be based on an individual person, because that person may change over time

• Backup individuals should be identified in the event that the primary is unavailable

• Job descriptions should be updated to include these responsibilities

• Performance and job evaluation criteria should include measurements on BCP work

• Job descriptions should include estimated time commitment on BCP tasks and estimated schedules

Plan forMaintenance

IT Schedules

BCP Policies

Formal Roles

Job Descriptions

BudgetaryPlanning


55

Phase 4 - Budgetary Planning• If Business Continuity Planning is to

succeed, money must be allocated

• The BCP consultant should work with the organization to help with:Identification of current Business Continuity

Planning budget Analysis of suitability of current budget and

suggestions for improvementDocument justifications for the creation or

enhancement of Business Continuity Planning budgets if necessary

Plan forMaintenance

IT Schedules

BCP Policies

Formal Roles

Job Descriptions

BudgetaryPlanning


56

Phase 4 - Change Control Systems

• Once the plan is in place, it must be modified in a controlled manner

• For this reason, some type of change control system must be implemented to ensure that the plan isn’t adversely modified in the future: Identifying when to re-evaluate the current environment

(performing a new risk assessment when significant changes have been made to the environment)

Identifying when to revise detailed build and recovery plans

Identify the system by which changes to the plan will be proposed, evaluated, approved and adopted

Plan forMaintenance

IT Schedules

BCP Policies

Formal Roles

Job Descriptions

BudgetaryPlanning


57

Phase 4 - Ongoing Maintenance• At this point, the bulk of the work has

been done and the organization is in “maintenance mode”

• The BCP group must regularly meet and evaluate the written plan

• Is it still up to date? Are changes necessary?

• Are major changes planned in the future? If so, what can you do to plan for them in the Business Continuity Plan?

• If changes are required, they should be proposed and go through the established change control system

Regularlyreconvene theBCP planning

committee,compare the

written plan tothe current

environment,propose

changes asneeded to

update

OngoingMaintenance

As needed, updatethe plan using

established changecontrol systems.For significant

changes to the plan,proceed to Phase 1

for a RiskAssessment. Forsmaller updates,

proceed to Phase 2to update individual

components

Update Plan

58

Discussion

Mark LachnietSr. Security EngineerAnalysts International

[email protected]

analysts international business continuity planning v1.01 10-2002

Documents

approach slide

existing business

risk assessment slide

entire business

business continuity

business impact analysis

disaster recovery planning

recovery procedures