Data Center Data Center Data Center Data Center Relocation…Take One!Relocation…Take One!

Joseph E. Ford, RCDDp ,Craig A. Lowe, RCDD/OSP,LEED AP

Robert G. Hall, MCSDBala Consulting Engineers IncBala Consulting Engineers, Inc.

Target AudienceTarget Audience• One or more of the following:

– Data center over about 100 devices– Enterprise with 7x24 operations

Complex applications– Complex applications

• Growing well and improving IT management to matchto match– Setting goals– Measuring progressg p g– Managing to expectations

• Smaller DC need less time but the same steps

Project PlanProject Plan

Story PlotStory Plot• Start the Project

B ild N DC F ili

Beginning {

• Build New DC Facility• Information Transport System• Establish LAN / WAN / SAN

Middle {

• Move Equipment / Server Waves

End {

• Decommission Old DC• Close the ProjectEpilogue {

Project PlanProject PlanProject Management View• Charter• Discover Requirements• Design (and Budget)Design (and Budget)• Build• Execute

D i i• Decommission• Close out

Project CharterProject Charter

• Establish relationship with champion and PMEstablish relationship with champion and PM• Select core team members• Logistics and meeting scheduleLogistics and meeting schedule• Kickoff meeting• Start Project Plan and Presentation• Start Project Plan and Presentation

Pre-move PreparationPre-move Preparation• As you do discovery, you will find systems that

cannot be moved as they arecannot be moved as they are• This is a placeholder for projects that must be

monitored to assure they are on track for the movemonitored to assure they are on track for the move• Examples:

– Eliminate equipment that is too fragile to move (butEliminate equipment that is too fragile to move (but probably running very important programs that nobody living knows how to support)Systems that depend on hard coded IP addresses– Systems that depend on hard coded IP addresses that will not transfer to the new Data Center

– Applications that must be virtualized so they can l t i llmove electronically

Sidebar ProjectsSidebar Projects• These are subprojects that may develop in

addition to moving equipment• These are usually separate because they are

funded separately• Examples:p

– Confirming that the Operation Center will function as expectedP j t t d i f t h t b tibl– Projects to design future changes to be compatible with the new data center

Discovery for FacilityDiscovery for Facility• Establish Inventory of hardware unitsEstablish Inventory of hardware units• Extend into power, cooling, and cabling estimate• Extrapolate technology and growth changesExtrapolate technology and growth changes• Add summary to Presentation• Client signs off Basis of Design• Client signs off Basis of Design

Discovery for FacilityDiscovery for Facility• Assist with floor plan for cabinets and racks• Add floor plan to Presentation• Monitor MEPS design process of DC, MDF, IDF,

NOCNOC• Review fire suppression, security, and

environmental designenvironmental design• Review new building access for services and

docks• Add summary to Presentation• Help with budget approval(s)

Build and Commission FacilityBuild and Commission Facility• Monitor construction schedule• Guest internet at new DC• Confirm walls, floors, ceilings, electrical,

cooling, etc.• Commissioning• Add summary to Presentation

Information Transport SystemInformation Transport System• Discovery

– Add connection information to inventory • Design

– Design prototype cabinet(s) for topology modeling– Assist in topology decisions

D i t h d it h l ti– Design patch and switch elevations– Add prototype and elevations to Presentation

Manage bidding and leveling– Manage bidding and leveling– Help with budget and purchase orders

Single Line DiagramSingle Line Diagram• Pull

Schedule• Plan View

Data Center PlanData Center Plan• Tray Plan• Cabinet

Elevations

Information Transport SystemInformation Transport System• Build

– Coordinate schedules for cabinets, racks, trays, UTP and fiberC PDU i– Connect PDU to power strips

– Confirm demarcation pointsTest• Test– Test inventory and labeling of cables– Test switch patching– Test switch patching

Design WAN LAN SAN & StorageDesign WAN LAN SAN & Storage• WAN connects other business locations and the two

Data CentersData Centers• LAN and SAN must be operational at both locations

– If refreshing equipment or changing technology, you may b f th d t tbuy new for the new data center

– Otherwise, you may rent equipment for the old data center so you can move your equipment to the new data center

• Long lead times for WAN LAN & SAN Communication services

• New storage equipment may be burned in with theNew storage equipment may be burned in with the network

• Design must allow a couple of devices to provide a period of solid service before real hardware wavesperiod of solid service before real hardware waves

OK Where Are We?OK Where Are We?• HW inventory is verified with power and service

ti fi ld t dconnection fields noted• Location is chosen and facility changes are underway• Service connections are enumerated and ordered• Service connections are enumerated and ordered• Network topology has been selected• Information Transport System is at least being designed p y g g

and is probably out to bid• WAN, LAN and SAN design is settled and equipment

d dordered• Cabinets and racks are ordered along with power strips,

security and KVM equipmentsecurity and KVM equipment

OK Where Are We?OK Where Are We?

What Can Go Wrong From Here?What Can Go Wrong From Here?• You can shut down critical systems in error• You can forget part of a system• A piece of equipment may not come upp q p y p• You can lose a truck • You can try to move too much at one time• You can try to move too much at one time• You may not understand how a device connects

N t k dd i t t h li ti• Network addressing may not match applications

Discovery – Beyond HardwareDiscovery – Beyond Hardware• Get application blueprintingpp p g• Match applications to inventory• Identify 'Fragile Artifacts‘• Identify Fragile Artifacts• Identify unique parts risks• Map telephony• Check both the routes and detours

– Walk and ride and walk

What’s an Application Blue Print?What s an Application Blue Print?• An application

centered view of how devices connect and service business transactions

• Includes:– Devices (on

inventory)– IP addresses and

network rules– OS/DB versions

and patch levels– Virtual and

physical attributesphysical attributes– Recovery and DR

concepts

Collecting Application BlueprintsCollecting Application Blueprints• AppOwner: The Application Owner

– May be in one of several places in IT or be an internal customer depending on the enterprise– Think about who approves scope changes in the application

• BPT: The BluePrint Team– Members from Architecture, Application Development, DBA, SA, networking and business relationship– Think about all groups that make a change to install or change an application– Think about who will be the scribe

Prepare• About 2 weeks• Identify AppOwner

Capture• Interview (1-4hr)

• White board

Validate• BPT sends to

interviewee

SignOff

• BPT sends

Distribute

• BPT providesy pp• Request an

interview• AppOwner agrees

on resources• Invite the team and

work out invitation

• Capture data• Identify gaps• Assign tasks

• Follow-up (1-2day)• BPT documents &

formats data

• Interviewee reviews diagrams and data sheets (2-3day)

• Interviewee &BPT reconcile changes

proposed diagram and data sheets to AppOwner

• AppOwner reviews for accuracy (1 week)

• BPT provides copies to Operations, Application Development, Architecture, and Operations teamswork out invitation

countersformats data

• BPT creates diagrams

)• AppOwner signs off

Operations teams

Iterations

Applications Move TogetherApplications Move Together• Cross reference devices and applications

– Every device should have known applications– Every applications’ devices should be known

• Reduce unnecessary cross dependency– Migrate storage, over time an enterprise tends to

d l id b f li ti h idevelop spider webs of applications sharing resources with each other

Design the Actual RelocationDesign the Actual Relocation• Decide on optimal waves size• Plan physical and virtual servers in waves• Publish the wave plans• Arrange for smart hands and movers• Assign locations in new DCg• Assign all patching• Print initial copies of documents for wavesPrint initial copies of documents for waves

Design the WavesDesign the Waves• Rehearsal

– Make sure players understand the processp y p– Include essential services:

Domain controller, DNS, etc.Base equipment for a VM farm

• Low hanging fruit– Power and IP (UTP or Fiber) only– Test case on each serviceTest case on each service

• Complex– SAN connections, telephony, unique parts

• The rest of the story– Fragile artifacts, redundant gear, retries– But do NOT save the worst for last

Preparing for a WavePreparing for a Wave• Confirm pre-test compliance

• Schedule a fix and test– OS patches and software upgrades

IP dd d t l h h– IP address and telephony changes– Power down and power up

• Freeze machine• Freeze machine

• Schedule backup, SA, DBA, testers

C fi d d t hi• Confirm core and edge patching

• Update, print and post (communicate!)Machine sheets posters elevations– Machine sheets, posters, elevations

Doing a WaveDoing a Wave• Day of move:

– Team checks– Go/no-go meeting

W h i d bl k illWeather, emergencies, road blocks, illness, etc.– Food

Z h• Zero hour:– Manage “war” rooms

Shared video and telephone conference lines– Shared video and telephone conference lines– Track every system

Doing a Wave - ContinuedDoing a Wave - Continued• Confirm backups• Management approval to power down• Ping to confirm downPing to confirm down• Uncable, unrack, shock sensor, load

L d t fi h t k• Loadmaster confirms each truck• Unload, check for shocks, rack, patch• Power up in sequence• Ping to confirm upPing to confirm up

Wave Need: Machine SheetsWave Need: Machine Sheets• One sheet for each device

– Tape to device early on move dayTravels with device– Travels with device

• Border indicates wave and truck• Large print has

– Make, model and unique identifier, q– Old rack and RTU number– New rack and RU number– Date truck and bin number

• Accurate rendition of attachment points• Accurate rendition of attachment points • Graphic and table of patch information

– IP patches with color, length and port– ILO & console patches with same– Fiber patches – Power cord length, receptacles and ends– Telephone lines

• Text boxes for overweight or special g pconnections like USB or heartbeats

• Verified by smart-hands before and after

Wave Need: PostersWave Need: Posters• Pull order at old

itsite

• Mount order at new site

• Red to green criticality

• Group lines

• Post at least t itwo copies

• “Red” system out last; in 1stout last; in 1for shortest down time

Wave Need: ElevationsWave Need: Elevations• New Data

Center onlyCenter only• Elevations by

row posted on peach row at new site– Color coded by y

waves (avoid red)

• Single cabinet strips postedstrips posted front and rear– Current wave

whitewhite

Testing Behind a WaveTesting Behind a Wave• Confirmation from application test team

– Regular tests approach 95% accuracy– A passed test means no errors were found– A failed test might not be a new target flawg g

• Be prepared to think• Morning after walk both DCMorning after walk both DC

– Check power – Check patching– Check placement

• Add summary of the wave to Presentation

Decommission Old DC and WrapDecommission Old DC and Wrap• We are not done yet – Set team expectations• Return leased and rented equipment• Dispose of other equipmentDispose of other equipment• Get leaser sign-off

Fi l C i ti i d• Final Communication issued• Close out meeting and documentation finalized

JEF@b l CAL@B l RGH@B lJEF@bala.com CAL@Bala.com RGH@Bala.com