university of new mexico “the agony & the ecstasy… the trials, tribulations & thrills of...

University of New Mexico

“The Agony & the Ecstasy…the Trials, Tribulations & Thrills

of an Enormous Project”

University of New Mexico

“The Agony & the Ecstasy…the Trials, Tribulations & Thrills

of an Enormous Project”

Mark ReynoldsMark Reynolds Associate DirectorAssociate Director

ITS Communications Network ITS Communications Network ServicesServices

The University of New Mexico: A City Within a CityThe University of New Mexico: A City Within a City

Founded in 1889 with 600 acres along old Route 66

27,000 students on primary campus; 7,000 at branch campuses

20,000 faculty and staff statewide including UNMH

Branch campuses in:– Gallup– Los Alamos– Taos– Valencia County

Health Science Center (HSC) University of New Mexico Hospital (UNMH)Health Science Center (HSC)

University of New Mexico Hospital (UNMH)

HSC, established in 1994, is largest academic health complex in the state

UNM Hospital (HSC's primary clinical component):– Consistently one of the 100 top-performing hospitals in the US– Primary teaching hospital for the

UNM School of Medicine– Home of the highly regarded UNM

Children's Hospital– New Mexico's only Level 1 Trauma

Center that treats more than 1,300 critically injured patients a year

UNM Children’s HospitalCommunications Challenges ($260 Million Project)

UNM Children’s HospitalCommunications Challenges ($260 Million Project)

480,000 square feet - six stories high 2000+ voice & data drops using Category Augmented 6 cable plant Telecommunications building in planned footprint

24,000 ports

19 PBX’s, multiple Key systems, DRUs

9,000 voicemail subscribers, 90 auto attendants

64 Automatic Call Distribution groups, 300 agents

2,200 cell phones, 4,000 pagers

What UNM Has…What UNM Has…

PBX SystemsSimplified Version

PBX SystemsSimplified Version

ESC

ON BATTLOAD ONBYPASSFAULT

Remote Systems - DRUs, NewBridges, Key Systems7-9-05

Pair Gain

Pair Gain

1 2 3

4 5 6

7 8 9

* 8 #

Load BatteryLineOn On

BatterySmartBoost

ReplaceBattery

Test

CSUs

CSU

Pair Gain

CSU

CSU

UNM PressIVS 2000

UNM Press (3721 Spirit Dr)

Norstar 8 x 24Key System

8 Phones, Ext. Only

Isleta Mental Health ClinicNorstar 8 x 24Key System

20 Phones, Ext. Only

Pediatrics (317 Commercial NE)


15 Phones, Ext. OnlyRed Ph. 837-6840

M&I(6900 Gonzales SW)

Electra Pro 1Key System

9 Phones Ext. OnlyRed Ph. 344-1080

M&I(1231 Candelaria NW)


11 Phones Ext. Only

Marron Hall (9)

Electra EliteKey System


Marron Hall (19)



Bandelier West (16)



ISR(2808 Central SE)


68 Phones, Ext. OnlyNEWBRIDGE (into Key System)

T1 Ckt: 59DHDA103155

UNM Ticket Ofc (1111 Univ. SE)

DRU1 FB: 925-563915 Ports Avail.

ACD in Ticket Office

Rio Rancho(2600 American NW)

DRU (2)T1 Ckts: 59DHDA002383/84

OR 54HCXX002283/841FB: 897-3562

1, 14 Ports Avail.

Mitchell Hall (23)DRU

15 Ports Avail4 Pairs

Mesa Vista (56)DRU


Poison Control (234)DRU (2)

1FB: 272-10454 Ports Avail

7 PairsContinuing Ed.

(1634 University NE)

DRU (5)1FB: 277-6851

2,4,23,23,23 Ports Avail.26 Pairs Each

Hope House(1816 Sigma Chi NE)

DRUT1 Ckt: 59DHDA 106124

8 Ports Avail

Humanities (81)DRU


Scholes Hall (10)DRU (3)

9, 7, 15 Ports Avail8 Pairs Ea.

Farris Eng. (119)DRU (2)


Student Services (85)DRU (2)

10, 17 Ports Avail.42 Pairs Ea.

256

Load BatteryLineOn On

BatterySmartBoost

ReplaceBattery

Test

Academy Clinic(7801 Academy)

DRU T1: 54HCXX001923


Casa EsperanzaKey SystemNorstar 8x24


Eye Clinic(1600 Univ)

DRU (2)T1s: 59DHXX000371/72

1FB: 242-2947

ASIC/OSIS(1213 Univ)

DRU1FB: 247-025113 Ports Avail

Nursing & PharmacyKey SystemElectra Pro I

8 Phones, Ext. Only

SEH Clinic(302 A San Pablo)

DRUT1: 59DHDA105853

1FB: 254-7791

Eye Clinic (269)DRU

1FB: 272-095916 Ports Avail.

EMS Academy(2700 Yale SE)

DRU T1: 59DHDA967175


DMAT/CREST(2704 Yale)

DRUT1: 59DHXX000340


Orthopedics(1101-5 Medical Arts NE)

DRU (2) T1 Ckts: 59DHDA104300/301

1FB: 242-26613, 15 Ports Avail.

Office of Evaluation(2730 San Pedro NE)


1FB: 884-3986

MHC(2600 Menaul NE)

DRU (2 - Pair Gain & Direct)

1FB: 272-280010, 18 Ports Avail.

33 Pairs Ea.

CRTC

CTH(1127 Univ. NE)

DRU 1FB: 272-476811 Ports Avail

8 Pair

Truman Clinic(625 Truman NE)

DRU (2)T1 Ckts: 54HCGS967518/558 GST Ids: NM.HCE.60032/33

0, 13 Ports Avail.

Corrales/Westside(4808 McMahon)



Dermatology ClinicDRU

T1: 59DHXX0003121FB: 247-144018 Ports Avail.

Walter Clinic(1500 Walter)

DRUT1: 54HCGS001849


Airport/Life Guard(2505 Clark Carr Loop SE)

DRUT1 Ckt: 59DHXX000320


1720 Louisiana NEDRU(2)

HSC Nat.Amer T1Ckt: 59DHDA556067

1FB: 232-636012 Ports Avail.HSC Geriatrics

T1 Ckt: 59DHXX0003351FB: 232-638023 Ports Avail

UPA (PALS)(1650 Univ.)

DRUT1: 54HCGS967629


Pair Gain

Pair Gain

Speech & Hearing (2001 Lomas NE)

DRUT1 Ckt: 59DHXX000375

1FB: 243-0692

Pair Gain

CSU

Latin Amer. Inst.Key System

1A2KSU6 Phones, Ext. Only

Latin Amer. DatabaseKey System

1A2KSU3 Phones, Ext. Only

Casaa Runaway(325 Washington SE)

NEWBRIDGET1: HCGS001852

1FB: 247-7451

Architecture(720 Vassar)

NEWBRIDGET1: 59DHDC100973

1FB: 4-119410 Avail. Ports

Mitchell HallPair Gain

NEWBRIDGE1FB: 1347/134819 Ports Avail

Elks Lodge(1642 Univ.)

NEWBRIDGET1: 54HCGS971716 &

50HMUC08606412 Ports Avail.

Dental Residency(4216 Louisiana)

DRUT1: 59DHXX00366


NB

ACTDRU

404 San Mateo1FB: 265-5798

256 DRU (22 cards and DRU’s) CRTC DRU (25 cards and DRU’s)

1101-4 Medical Arts

5901 Zuni

First StepWhat to do with the University’s Main Central Office PBX?

First StepWhat to do with the University’s Main Central Office PBX?

Option 1: Move 11,000 port PBX to another location (estimate $10 million)

This would require:– New building and power plant

– New PBX

– Move all copper, fiber and carrier connections

Decision: Expense and time unacceptable to Hospital

Second option: Leave the PBX where it is & protect the PBX building– Hospital estimated $400,000 to fortify the PBX building

– Actual final cost was $2 million plus new office building

Decision: Work with design team to protect the University’s communication systems

Next Steps - Survival PlansNext Steps - Survival Plans

Identify risks and probability of system failures

Develop a mitigation & contingency plan

Develop a disaster recovery / incident action plan

Make improvements to construction plans

Develop detailed work plan

Perform audits of switch-room, electrical and cable plant

Rehabilitate cable vault

Replace voice processing system

Next Steps – Survival Plans Next Steps – Survival Plans

Reroute carrier facilities, negotiate with LEC for two additional POPs & make Hospital PBX independent

Reroute cell phone services

Reroute alarm / emergency services

Develop emergency services agreement with outside firm

Create position to oversee protection of switch room and manage cable plant

Next Steps - Telephone Planning StudyNext Steps - Telephone Planning Study

Five year plan with input from stakeholders

Ensure high quality customer service and “state of industry” systems

Enable cost controls

Ensure self-financial support

Plan for technical and operational convergence

Move thoughtfully toward convergence, evolve with current manufacturer (NEC)

Use multi-vendor solutions where practical

Plan / carry out VOIP trials

Perform cost allocation study to determine true costs

Develop rates and revenue strategies

Existing PBX roomExisting PBX room

Construction BeginsConstruction Begins

Construct new office building for telecom staff

Install and cutover new voicemail system

Demolish building around the PBX room without structural damage

Keep PBX room sealed, provide positive pressure to keep dust out and build vestibules at both entrances

Carrier changes (two additional POP’s)

Power design and changes

HVAC system - “clean” the duct work (original plan)

Remove copper cable feeding existing telecom building

Move the carrier fiber paths that feed the PBX room

Move two cell sites

Overcome public utility power pole complications

Remove ceiling tiles to avoid equipment damage when removing copper plant

Design new lighting fixture layout

Decommission Halon system and install FM200 system

– Halon dump accident– New piping system

Construction ContinuesConstruction Continues

Construction ContinuesConstruction Continues

Remove existing HVAC system which consisted of:– Indoor Liebert Unit– Ceiling Unit for backup

Install two 20 ton units using “chilled water” and DX backup – Hole in roof for exhaust of portable systems– Portable units in room to keep temperature

at 68 degrees with 25% humidity– Install new duct system

Install new generator to replace existing one – New grounding system – Power within the PBX room rerouted and

re-cabled

Construction Time Lapse Video Construction Time Lapse Video

Biggest Construction ConcernsBiggest Construction Concerns

Vibration and Core DrillingVibration and Core Drilling

Beam InstallationBeam Installation

The Best Laid Plans….The Best Laid Plans….

Dust– No vestibules built, see daylight through

walls and roof– PBX equipment and room fill with dust– Heat and aging PBX concerns—could fail– Engaged external and internal

professional services to document Contractor

– Installed devices to assist in damage control, but the damage was already done

SolutionSolution

Bonding authority received for switch replacement and upgrades--$5 million

RFP developed

– Replace the 20+ year old, 11,000 port main PBX

– Upgrade the Hospital 10,000 port PBX

– Various ancillary system and PBX upgrades

– Outsource the maintenance and installation functions

Telephone Planning Study recommendations followed

Next StepsNext Steps

RFP completed– Negotiation & award of contract– Upgrade all systems, replace

sets not ADA compliant– Transition maintenance group from

existing contractor to new contractor

Installation of fiber paths to the University, Hospital, and data center with full fail over


Establish PBX upgrade/replacement project and communications plan

Maintain support for work orders and trouble tickets during PBX implementations

Build new Hybrid PBX and distribution frame within same room as existing PBX (space limitations)

Tie hybrid PBX’s to existing PBX for migration strategy

Design and install temporary emergency PBX for the hospital PBX upgrade


Customer service responsible for “getting the word out”

Communication methods included:

– Flyers posted on entry doors and throughout affected buildings

– Telephone calls to key department representatives

– Email to building coordinators and mass emails

– UNM News Minute update

– Schedules, FAQ’s and other documentation posted on the web

– Posting on the outage notification whiteboard

Communication process changed as the criticality of the situation increased.

The Trouble Begins!The Trouble Begins!

Roof– Roof leaked during snow and rain fall– Installed sump pumps with alarm

system – New roof and drains

HVAC system– 20 Ton units come on line, backup

system does not work

The Trouble Intensifies!The Trouble Intensifies!

Our WORST FEAR!– PBX starts to fail

• No dial tone, call forwarding, auth code failures

– Cannot shut down the PBX– Weekend – PBX fails but comes back– New PBX is working but not all customers have

been moved over– Fast track the project

• Cut over 4000 pair a day

• 2-3,000 end users affected daily

• Work day and night to cut-over

• Move E911, voicemail & special circuits

• Affected Hospital auto attendant & clinics

New University Switch RoomNew University Switch Room

UNMHeart – Courage - Wisdom

UNMHeart – Courage - Wisdom

Thank You! Thank You!

university of new mexico “the agony & the ecstasy… the trials, tribulations & thrills of...

Documents