university of washingtoncomputing & communications networking update terry gray director,...

University of Washington Computing & Communications

Networking Update

Terry GrayDirector, Networks & Distributed Computing

University of Washington

UW Medicine IT Steering Committee16 January 2004

20 February 2004


Outline

• In our last episode…– Context– Expanded Partnership– Recent Problems

• Today– Systemic Problems and Progress– Network Security Chronology– Design Issues


Context: A Perfect Storm Increased dependency on network apps Decreased tolerance for outages Decades of deferred maintenance... Inadequate infrastructure investment Some old/unfortunate design decisions Some extraordinarily fragile applications Fragmented host management Increasingly hostile security environment Increasing legal/regulatory liability Importance of research/clinical leverage


Key Elements of the Partnership

Changed: C&C now responsible for... In-building network implementation and

operational support for med ctrs, clinics Med center network design “for real”

Not Changed: C&C still responsible for... Network backbone, routers Regional and Internet connectivity SoM and Health Sciences networking


Why the Partnership Makes Sense Consistency, interoperability, manageability Leverage C&C networking expertise Clinical/research hi-performance network needs 24x7 Network Operations Center (NOC) Advanced network management tools Avoid design/build organizational conflicts Beyond the network...

hope to share distributed system architecture and network computing expertise


Recent Problems Oct 29: Partial router failure reveals escalation

procedure problems Oct 30: Security breach triggers connectivity and

server problems Nov 12: 13 minute power outage triggers extended

server outage Dec 12: Router upgrade uncovers wiring error, which

triggers multicast storm

(None of these were related to the network transition, save perhaps timing of #4)


System Elements

Environmentals (Power, A/C, Physical Security) Network Client Workstations Servers Applications Personnel, Procedures, Policy, and Architecture

Failures at one level can trigger problems at another level; need Total System perspective


Reasonable Questions

What’s up with C&C’s alarm system vendor? If power was out for only 14 minutes, why was service

out for multiple hours? What can we say about an app so fragile that a net

interruption of a few seconds requires a server reboot? What can we say about thin clients built on top of

thick (WinXP) operating systems? What can we say about a network where one wiring

fault can disable most of the net?


Systemic Problems and Progress


Systemic Network Problems(NB: these pre-date Tom et al)

Old infrastructure (e.g cat 3 wire) Non-supportable technologies (e.g. FDDI) Non-supportable (non-geographic) topology Expensive shortcuts (e.g. cat5 mis-terminated) Security based on individual IP addresses Subnets with clients and critical servers Documentation deficiency

Contact database Device location database Critical device registry


Systemic General Problems

Ever-increasing system complexity, dependencies Departmental autonomy Un-controlled hosts Un-reliable power and A/C in equipment rooms No net-oriented application procurement standards

Are HA and DRBR expectations realistic? Are backup plans workable?


Some Numbers

UW Total(incl UWMedicine)

HealthSciences(incl SoM)

MedicalCenters

Subnets 1022 52 145

Devices 70,000 >8,000 10,000


Network Device Growth

Note: Most dips reflect lower summer use; last one is a measurement anomaly


Network Traffic Growth (linear)


Network Traffic Growth (log)


Near-term Progress and Plans Agreement on standard maintenance window Created “Top 10” list --creeping to Top 20 :) Static addressing work-around (success!) FDDI, VLAN elimination Subnet splits/upgrades (1500 computers) Equipment upgrades Router consolidation, dedicated subnets, separate med

center backbone Equipment, outlet location database updates Initial wireless deployment


Design Review and Cost Estimates

Biggest cost: physical infrastructure & wireplant upgrades

NetVersant engaged for cost estimation project Cisco engaged for network architecture review We recommend similar reliability/design

assessment for servers, apps & procedures


Design Issues


Design Tradeoffs

Networks = Connectivity; Security = Isolation Fault Zone size vs. Economy/Simplicity Reliability vs. Complexity Prevention vs. (Fast) Remediation Security vs. Supportability vs. Functionality

Differences in NetSec approaches relate to: Balancing priorities (security vs. ops vs. function) Local technical and institutional feasibility


Tradeoff Examples• Defense-in-depth conjecture (for N layers)

– Security: MTTE (exploit) N**2

– Functionality: MTTI (innovation) N**2

– Supportability: MTTR (repair) N**2

• Perimeter Protection Paradox (for D devices)– Firewall value D– Firewall effectiveness 1 / D

• Border blocking criteria– Threat can’t reasonably be addressed at edge– Won’t harm network (performance, stateless block)– Widespread consensus to do it

• Security by IP address


Network Security Credo

• Focus first on the edge(Perimeter Protection Paradox)

• Add defense-in-depth as needed

• Keep it simple (e.g. Network Utility Model)

• But not too simple (e.g. offer some policy choice)

• Avoid – one-size-fits-all policies– cost-shifting from “guilty” to “innocent”– confusing users and techs (“broken by design”)


Preserving the Net Utility Model

• What is it?• Why important?• Incompatible with perimeter security?• Too late to save?• NUM-preserving perimeter defense

– Logical Firewalls– Project 172

• Foiled by static IP addressing…– Requires all hosts be reconfigured


Lines of Defense

• Network isolation for critical services.

• Host integrity. (Make the OS is net-safe.)

• Host perimeter. (Add host firewalling)

• Server sanctuary perimeter.

• Network perimeter defense.

• Real-time attack detection and containment.


Network Security Chronology• 1990: Five anti-interoperable networks• 1994: Nebula shows network utility model viable• 1998: Defined border blocking policy• 2000: Published Network Security Credo• 2000: Added source address spoof filters• 2000: Proposed med ctr network zone• 2000: Proposed server sanctuaries• 2001: Ban clear-text passwords on C&C systems• 2001: Proposed pervasive host firewalls• 2001: Developed logical firewall solution• 2002: Developed Project-172 solution• 2003: Slammer, Blaster… death of the Internet• 2003: Developed flex-net architecture


Next-Gen Network Architecture Parallel networks; more redundancy Supportable (geographic) topology Med center subnets = separate backbone zone Perimeter, sanctuary, and end-point defense Higher performance High-availability strategies

Workstations spread across independent nets Redundant routers Dual-homed servers


Success Metrics

Tom’s Nobody gets hurt Nobody goes to jail

Terry’s “Works fine, lasts a long time” Low ROI (Risk Of Interruption)

Steve’s Four Nines or bust!


Success Metrics II

We all want: High MTTF, Performance and Function Low MTTR and support cost

The art is to balance those conflicting goals we are jugglers and technology actuaries


Success Metrics III

How many nines? Problem one: what to measure?

How do you reduce behavior of a complex net to a single number?

Difficult for either uptime or utilization metrics

Problem two: data networks are not like phone or power services… Imagine if phones could assume anyone’s number Or place a million calls per second!


Concerns, Future Challenges Mitigating impact of closed networking:

Needs of the many vs. needs of the few Pressure to make network topology match administrative boundaries Complex access lists False sense of security Increased MTTR

Next-generation threats: firewalls won’t help Security vs. High-Performance Wireless Balancing innovation, operations, & security


Lessons Five 9s is hard (unless we only attach phones?) Even host firewalls don’t guarantee safety Perimeter firewalls may increase user confusion, MTTR Nebula existence proof: security in an open network Even so… defense-in-depth is a Good Thing It only takes one compromise inside to defeat a firewall Controlling net devices is hard --hublets, wireless The cost of static IP configuration is very high Net reliability & host security are inextricably linked Never underestimate non-technical barriers to progress


Questions? Comments?

university of washingtoncomputing & communications networking update terry gray director,...

Documents

network apps

network backbone

network implementation

network transition

progress slide

server problems

partnership recent problems

total system perspective