1

At Katrina’s EdgeAt Katrina’s Edge

One Campus’ StoryOne Campus’ Story

Brian D. VossBrian D. VossLouisiana State UniversityLouisiana State University Contributions from: Sheri ThompsonContributions from: Sheri Thompson

2

Acknowledging those most affected

• John Lawson, my colleague from Tulane and Jim Burgard, my colleague from University of New Orleans

• Our IT colleagues at Xavier, Loyola, Dillard, Southern (NO), and Southeastern, in the Louisiana Technical and Community Colleges, and those in the states of Mississippi, Alabama, and Texas impacted by Katrina and Rita.

• Clifford Woodruff at Lamar University, who rode out Rita, has some excellent insights to share; ask him

• I was on this disaster’s edge; they were at its center.

3

A little level setting• Louisiana State University is Louisiana’s Flagship public

university campus located on the Southeast side of the capital city of Baton Rouge, which is about 75 miles northwest of New Orleans

– ~30,000 students

– ~4,500 faculty/staff

• Both major storms ‘missed’ the Baton Rouge campus

– Katrina passed about 80 miles to the East of the campus

– Rita passed about 120 miles to the West of the campus

• No part of LSU (nor Baton Rouge) was flooded nor extensively damaged

• In short, we were impacted to a lessened degree by the storms; but we were impacted by their aftermaths

4

Hurricane 101• A Midwestern transplant learns about hurricanes

– A CIO gets an early lecture on coastal erosion, river delta disintegration, and hurricanes

– Saffir-Simpson Scale• Wind-speed gets the headline, storm surge is the unseen destroyer • Pressure = strength (the lower, the stronger)

– The effect of ‘deep warm water pools’– Hurricanes hate dry air– West side good, East side bad

• Legendary Stories of Camille, Betsy, Andrew & Ivan• Arlene, Cindy & Dennis – 2005 warm-ups

– Bookmarking NOAA’s National Hurricane Center; LSU Hurricane Center

• Katrina & Rita … and then Wilma; a busy season

5

How LSU Prepares for a Hurricane

• ~72 hours in advance, we begin to watch– LSU Hurricane Researchers work models, predominantly

focusing on storm surge– Leadership begins regular meetings

• ~48 hours in advance, we begin to make decisions– To close or not to close?– Baton Rouge’s main concern is evacuee traffic

• ~24 hours in advance, we do final prep– Campus battens down the hatches– Critical Personnel Only– Computing Center implements ‘Hurricane Status’

• Top off generator diesel tanks• Schedule critical personnel• Run final back-ups and quiescence of non-critical systems

6

Tropical Storm Arlene~990mB 60mph June 11

7

Tropical Storm Cindy~997mB 60mph July 6

8

Cindy – No big deal

• Relaxed meeting of campus leadership• “Phoned-in” briefing from LSU Hurricane

Researchers– No worries – this one will miss us and not be that bad

• Cindy sweeps “wide right” and Baton Rouge gets a nice, sunny, warm June day– Florida panhandle and Alabama & Mississippi gulf

coasts get their first taste of the 2005 season– New Orleans, just to the west of the ‘eye’ gets some

heavy weather, but nothing it couldn’t handle

9

Hurricane Dennis~930mB 125mph July 6

10

Dennis – first serious escalation

• Leadership Meetings Called– Two meetings – at 72 and 48 hours– 24-hour meeting is cancelled

• Leadership gets briefing from LSU Hurricane Researchers– “This 500-lb Linebacker …”

• We go on ‘standby’ but don’t “pull the trigger” on any closings

• Dennis sweeps “wide right” and Baton Rouge gets a nice, sunny, breezy hot July day– Panhandle gets punched … again

11

Hurricane Katrina~900mB 165mph August 28

12

Hurricane Katrina~902mB 165mph August 28

Thursday August 25

Sunday August 28

Tuesday August 23

Saturday August 27

Initial Projected Path

13

Katrina – Suddenly Bearing Down

• Saturday August 27 – dawns as the day of the LSU Faculty/Staff Golf Summer Tournament– Chancellor O’Keefe had been out of town– Katrina hits Florida but was to turn hard right and pop

the Panhandle (again) … she doesn’t turn– Cell phones start going off mid-round

• First leadership meeting 5pm Saturday 8/27; chancellor phones in from the road trying to drive back– LSU had started classes the previous week; dorms

full– LSU calls off classes for Monday (day storm to make

landfall); closes University for faculty/staff

14

Katrina – Suddenly Bearing Down

• Second leadership meeting Sunday 8am– Decision made to close Tuesday 8/30 as well– Faculty/Staff back on Wednesday 8/31;

Classes resume 9/1

• All other prep advanced – we’re ready• Katrina becomes a Category-5; sustained

winds over 160mph, gusts near 170. New Orleans is dead-on in the projected path– Cat-4 at landfall, but with strong Cat-5 storm

surge

15

Sunday, August 28Landfall at the CIO’s House

• Sunday AM – clear, bright, warm– Bottled Water, canned/boxed food, batteries, ‘Armageddon Radio’– A dip in the pool anyone?– Inland Hurricane Wind-Force Warning

• Sunday 2:54PM – first bands hit BTR• Sunday 6:40PM – Tropical storm-force winds gust in BTR

(>35mph)• Sunday 10:15PM – Off to bed• Monday 5:57AM – Up – winds howl

– Half a pot of coffee and then no power– First LA landfall around this time

• Pretty Scary Morning – on ‘the good side’ of the Hurricane– Wind but little rain (<2 inches in 24 hours)

• Winds begin to subside around noon but are steady• Afternoon is windy – still getting gusts >40MPH• By 4PM it seems to be over, if still quite windy (25-35MPH)

16

Monday, August 29“We’ve been hit, but we made it”

• Katrina’s Immediate Aftermath– Lower LA Parishes swamped by storm surge; no real word out– Parts of New Orleans flooded, at least one levee over-topped,

but city seems to have survived– SE Louisiana devastated by winds/rain– Mississippi seems hardest hit

• Monday 5pm Meeting at LSUPD Station– We Survived … just a little damage– Data Center Lost power but fallover to back-up worked perfectly– Everything Looks Good to Go for Tuesday clean-up, Wednesday

start-up, and Thursday-as-usual– Mood light and jovial– Power restored to campus ~6:15pm– LSU’s HPC resources running full-bore, analyzing storm data

17

Tuesday 8/30“The Bowl is filling up”

• Reports confirming catastrophic levee failure in New Orleans occur at 1:30AM CDT

• By mid-afternoon, >80% of New Orleans is under water

• Evacuees/Refugees • LSU contacted about expanding routine special

evacuee facilities into a broader purpose– Medical Triage (Pete Maravich Assembly Center)– Special Needs Facility (Field House)– First IT needs – Phones, phones and more phones– The first Five O’clock Meeting

18

Wednesday 8/31“Somebody’s surely going to take charge today”

• PMAC & FH Ramp Up– Flow of wounded, sick, & displaced increases dramatically

• Agencies, Doctors, Volunteers show up– More Phones, data connections, computers

– Coordination Lacking – LSU steps up

• Two-a-days commence

• Choppers! – It’s like M*A*S*H

• Sirens wail – streets not disturbingly deserted are disturbingly filled with uniformed men and medical teams, and sick, wounded, and displaced

• LSU postpones return of students until 9/6

• The Superdome

• UNO staffer shows up at Frey CC – has their webspace back-up on his laptop

19

Thursday 9/1“We’re not following the book – we’re writing it”

• Mayor Nagin’s passionate plea

• More agencies show up at LSU– Flow of patients and refugees increases

– Campus security issues

– Coordinating volunteers becomes a challenge

• LSU establishes Command Center– Coordinates information for students, and evacuees, as well as

directing resources to where they’re needed

• No one shows up to take charge – LSU assumes control in the vacuum

• First vendor donations and help arrive

• LSU’s first home football game (slated for Saturday) is postponed to later in the season

20

Friday 9/2“Will we be a university this Fall … or not?”

• Superdome gets relief – finally!

• Tensions mount – tempers flare• Flow shows no signs of dropping• PMAC has handled thousands• Leadership meetings

– Public Safety reports– Facilities reports– Student-life reports– Special Facilities Reports– Communications & University Relations Reports

• IT applications – needed at 8am, completed by noon, rendered unnecessary by 5pm

• More equipment!• KTR Student Enrollment commences

21

Saturday 9/3 & Sunday 9/4“Maybe you could put us up in a dorm room?”

• More IT help arrives – Microsoft and IBM send in disaster support teams– National Guard Convoy delivers 18-boxes of IT

support donations to LSU

• Information systems start to jell at PMAC/FH– FERPA/HIPAA

• LSU re-confirms that it will re-start classes – bumpy and rocky as it may be – on Tuesday following the Labor Day Holiday

• Donations, donations, donations• A Plan for Displaced Students

22

Labor Day 9/5“No sir! The Football stadium didn’t pass the fire

marshal’s inspection!”

• Students start to return to campus• AgCenter campus deals with

furry/feathery/scaly refugees• Moves/Adds/Changes• First signs of a let-up• Volunteers continue to staff the Hotline • Critical IT personnel ordered to take 12 hours off – NO

EXCEPTIONS!• “Tomorrow, we have our old jobs to do as well.”• LSU strives to retain its status as a university campus, in

light of its success as a disaster response facility

23

Tuesday 9/6 “I was never so glad to see a backpack in my life!”

• Classes resume!• KTR Enrollment & Registration intensifies• UNO CIO asks for and receives sanctuary• More calls inquiring about assistance• Moves/Adds/Changes – okay, now you’ve gotta

be kidding me!• Chancellor schedules ‘Town Meeting’

– Can we video stream that?

• Deans Monthly Meeting

24

9/7 through 9/20“We’re getting back to normal, whatever the hell that is.”

• PMAC & Field House eventually wind-down; Field House later temporarily re-opens for Hurricane Rita response– Over 6,000 patients treated at the facility during the first 7 days

• LSU completes special Katrina Enrollment; ~3000 new undergrads and >300 grads enroll from the impacted schools as ‘visiting students’

• All non-students (family members, first-responders, etc.) finally moved from dormitories; 100% capacity achieved with visiting students

• Temporary IT equipment dismantled; we lost some laptops, especially the three that went out on a Coast Guard S&R helicopter

• 5 O’clock Hurricane Response meetings ceased 9/12

25

Hurricane Rita~892mB 170mph September 23

26

Rita – Déjà Vu All Over Again?“All things considered, I'd rather be in Philadelphia.”

• I’d rather not be in Philadelphia– My agita had little to do with the I2/NLR merger

• Rita was supposed to bear South, but ‘sliced’ to the North– Campus Leadership meets again– Campus closes that Friday at 2:30 as rains and wind

begin; closes through Monday• While Rita was perhaps even more powerful

than Katrina, and Louisiana suffers yet again (along with NE Texas), the aftermath for LSU and Baton Rouge was not as significant

• We saw again what happens when a major American city (Houston this time) is threatened with destruction

27

Katrina’s Impacts on Louisiana Higher Ed

Basically, those in Katrina’s path were put temporarily out of business

• UNO/LSUHSC – portions of campus under water – classes resumed 10/10 at Jefferson Facility– Planning on resuming operations in NO Spring

Semester

• Tulane, Xavier, Dillard & Loyola– Campuses all impacted to one degree or another – Established temporary admin HQs elsewhere– Planning on resuming operations in NO Spring

Semester

28

Katrina’s Impacts on Louisiana Higher Ed

• Louisiana Technical Colleges & Community Colleges– Lost several NO-based campuses; serving students at

other campuses that were less (or not at all) impacted

• Louisiana State University– Became perhaps the most critical facility in support of

disaster relief/response in the State of Louisiana– Despite this, continued to function as the State’s

Flagship Research University; added over 3,000 temporary/displaced students; serve over 9,000 students from areas impacted by Katrina & Rita

29

Getting IT up and running

• Impacted institutions turned to technology– Websites first thing to go up to share info– Establishing e-mail communications– Use of online education (“DE”)

• Vendors stepping up to help– Donations, loans, discounts, financing

• Re-establishing operations in temporary locations

• Rebuilding campus infrastructure will be a huge challenge that is as-yet unclear

30

LSU’s IT support for UNO• Providing space for their staff (~30 people)

– Workstations, access, phones, facilities, supplies• Providing rack space in machine room for their recovery

servers• Assisted in restoration of their key web and e-mail

services• Providing commodity internet and internet2 connectivity

& bandwidth• Assisting with general sys-admin and data recovery

operations• Linked them up with our student e-mail outsource

provider (Outblaze) for quick restoration of service to that community

• Providing support via LSU System Office for their administration

31

Help from the HE community

• EDUCAUSE and other sites help with information and ‘match-making’

• Other HE IT groups (Internet2, ACUTA, etc.) also seeking ways to help out

• Regional groups (like SURA)• CAVEATS:

– Dealing with more than a loss of IT– Don’t kill them with kindness– Wait until asked– Recognize the logistics are nightmarish – Be interested in 2006

32

April 2006 Update• University of New Orleans

– Re-Opened for Spring 06 (enrollment ~70%)

• LSU Health Science Center– Temporarily moved educational & most research programs to Baton Rouge– Main facilities still being renovated/recovered– Dentistry moved to Baton Rouge – Expect to return to New Orleans Fall 06

• Tulane– Re-Opened for Spring 06 (enrollment ~77%)

• Dillard– Extensive damage to be addressed– Re-Opened temporarily in Hilton Riverside Hotel

• Xavier– Opened for Spring 06 (enrollment ~75%)

• Loyola– Re-Opened Spring 06 (enrollment ~90%)

33

Post-Katrina Disaster Recovery Thinking

• Traditional Disaster Recovery– What if my data center is lost

• Broader Disaster Recovery– What if my campus is lost– What if the city where my campus is located is lost

• Survivor Disaster Recovery– What if I’m fine … but everyone around me is not

34

Traditional Disaster RecoveryYou’re down, everything else is fine

• Do you have a workable DR plan?• Do you know where on campus you’ll go?• Did you take necessary back-ups and do you have them

ready to re-produce production files?• What vendors will you need to tap – and for what?• How will you quickly re-establish network connectivity?

Phone service? Web presence? E-mail? Mission critical information systems?

• Lessons:– It’s the data, stupid. Hardware can be replaced; data can not be– If you lose just your data center, the need for rapid response will be

acute and immediate; your institution can not operate without IT

35

Broader Disaster RecoveryYou and everyone around you is down

• Are your off-sites conveniently (and perhaps tragically) close?• Do you have arrangements to get key services restored at a

distance– Web, E-mail, Financial/HR, Student Information, CMS

• Hot-sites may be too much $$$$ – but can you find suitable raised floor/HVAC/power to ‘re-build’

• Can you support your administration “in exile?”– Internet access, computers, cell phones, e-mail, IM

• Is your ‘life-boat’ plan portable over larger distances?• Can you grab your key people? Can you care for them?• Lessons:

– People are your most key resource – but expect them to be burdened with other priorities

– Knowing what you’ll need to do and having it organized is more important than knowing all about ‘how’ you’ll do it when you get there (wherever ‘there’ might be)

36

Data Center Lifeboat

• Situation: What if we had very short notice (4-8 hours) notice of the need to abandon our data center/campus and set-up elsewhere (>50miles away)

• Goal #1: Re-establish some critical subset of services

• Goal #2: Support the re-establishment of some subset of university administration

37

Lifeboat

• Key things to recover:– Payroll/Financial Data– Web presence

• Splash/priority information screens

• As much content as possible– E-mail service for

faculty/staff/students– Portal interface– Student Information Systems– HR, Procurement Systems– CMS– What else?

• Budgets ($25K, $50K, $100K)

• Key things to address– Off-site storage of critical back-ups– Ability to ‘grab and go’ key data

and hardware– List of key hardware needed later

from vendors– Disaster Supplies Crate

• What would we put into an 8x12 truck for rapid evac?

– Equipment for a mobile or relocated university command post

• Laptops, radios, phones, etc.– Identify Key IT personnel

• Who does what w/back-up• “Scoop ‘em up”

– Where might we go?

38

Survivor Disaster RecoveryYou’re the last ones standing

• Dealing with unimaginable demands– Start imagining it

• Do you have a stock of equipment to set up a large support operation in short-order?– Networking gear, computers, cables, supplies, telephone service

• Value of a flexible and capable staff– They’ll see things no one should have to see

• Consider how you’ll do all this on top of your normal jobs, as campus life resumes and student enrollment increases

• How ready is your campus administration to take on the role of disaster response center?– Facilities, public safety/police, communications, academic affairs– Is the CEO (Chancellor, or President) prepared?

39

Lessons Learned at LSU• Have a good stock of networking equipment,

and mobile and desktop computing in the storeroom– Plan to raid campus labs & empty desks if need be

• Have strong relationships with key vendors• Great to have terrific, dedicated, service-oriented

people• Architectures count – how divisible are the

components? What’s removable as a component, and what’s too-tightly integrated?

• Be prepared to be flexible; adapt, improvise, overcome– Don’t be thin-skinned

40

Lessons Learned continued

• Good to “sit at the big table”– But know when to speak, and when not to – if you’ve

watched ‘Survivor’ you know what I mean

• Keep your friends close – and don’t have enemies

• Everything we’ve been saying about the strategic value of IT is valid; IT enables everything in the 21st Century– But even now – does HE administration ‘get it?’

• Disaster Recovery and Business Continuity Planning is not a luxury

41

Hurricane Season 2006 Starts in less than 60 days

• We’re working on a basic, nimble IT DR/BCP document – will be done by 5/31– Addressing all three forms of disaster– Requesting funds ~$500K

• Campus is constructing a ‘permanent’ EOC– Rather than relying upon hastily assembled one– Spending ~$150K to do it

• Chancellor put out a call for DR/BCP plans across key campus units

42

Thoughts 7 months after

In a disaster, do the rules – and the plan – go out the window?

• Isn’t the very nature of disaster its unpredictability?

• Can we ever plan for every possible event and circumstance?

• Will we have time to refer to a detailed disaster plan document?

43

What then should we do?

• Focus on the process of planning and not on the plan itself (so sayeth Capt. Joe Castillo USCG)

• Examine how we will position ourselves and our assets to be flexible in responding to a disaster

• Focus on knowing what will need to be done in the first stages, what we’ll need to do those things, and who will do them

• Plan to be flexible. Plan to improvise, adapt, and overcome

• Drill on these things

44

Again, What’s Important?

• Hardware and facilities can be replaced in the periods following a disaster

• Data is the primary focus of what you need to be prepared to restore and the basis of continuity

• People are your most key asset

45

“We had a failure of imagination.”

• Lessons from NASA; Apollo 1 and Columbia Shuttle disasters

• We need to imagine the questions first so that we can find the answers

• We need to – as a community – seek answers together

– How can we leverage national cyberinfrastructure?

– Individual arrangements versus broader approaches

• How seriously do you, as CIO, take the strategic nature of IT? How about your administration?

– IT truly is an enabler of everything we do now

– Are our people ready to be First Responders?

46

Those who cannot learn from history are doomed to repeat it.

-George Santayana

• CIOs can no longer say they can’t imagine what could happen – because it just did.

– Or an earthquake, or a tsunami, or a terrorist attack, or an accident, or a pandemic

• Next time, you may not be watching it on CNN – you may be living it

• Now is the time to think, plan, and take action – later it will be too late

47

At Katrina’s EdgeBrian D. Voss

Chief Information OfficerLouisiana State University

bvoss@lsu.edu Special Thanks: Charlie, Cindy, Ric, Andy, John, Sheri, Robin, Brian, Byron, Sean, Scott, Susan, Sara, Ralph, and the entire staff of ITS, OCIO, and LOUIS.

Geaux IT@LSU!