cisco hight availability enterprise network design

8/8/2019 Cisco Hight Availability Enterprise Network Design

1/58

1 1999, Cisco Systems, Inc.5050911_04F9_c3

HighHigh --AvailabilityAvailabilityEnterprise NetworkEnterprise Network

DesignDesign

havilandhaviland @[email protected]


2/58

25050911_04F9_c3 1999, Cisco Systems, Inc.

Staying On TargetStaying On TargetHA FocusHA Focus vsvs Distractions!Distractions!

Flat networksare easierbeware!

Fivenines isjob one!

Inheritedcomplexityhard to purge

The latest

cool stuffolder is morestable

Varietyof vendors,protocols,designs, etc.

Featurerichlets use allthe knobs!

Change is hard,sometimes $$$


3/58


HA Features of the Catalyst 6500HA Features of the Catalyst 6500Consider for Backbones & Server FarmsConsider for Backbones & Server Farms

Fabric Redundancyswitch fabric module

in CatOS 6.1

Supervisor Redundancy

HA feature in CatOS 5.4.1stateful recovery

image versioning on the fly

MSFC Redundancyconfig-sync feature

IOS 12.1.3 CatOS 6.1HSRP pair


4/5845050911_04F9_c3 1999, Cisco Systems, Inc.

Thinking Outside the BoxThinking Outside the Box

For HA/HP designoutside the box

the logical designis critical network features& protocols geophysicaldiversity is powerful

Inside:HA,RAID,UPS,

MTBF,etc.



DramatisDramatis PersonaePersonaeOur Cast of SymbolsOur Cast of Symbols

LinksGE, DPT, SONET, etc.

L2 switchingL2 forwarding in hardware L3 switching

L3/L2 forwarding in hardware Routing

L3 forwarding (SW or HW)

Control plane = IOSrouting protocols & features

QoS where required Application intelligence

Catalyst 4000

Cisco 7500 Cisco 12000

Catalyst 6500

GigE Channel



ClientBlocks

Distribution L3

Access L2

HA Gigabit Campus Architecturesurvivable modules + survivable backbone

Backbone

ServerBlock

Server Farm

Distribution L3

Access L2E or FE PortGE or GEC

Ethernet or ATMLayer 2 or Layer 3

Definethe missioncritical partsfirst!



High Availability DesignHigh Availability DesignWhy aWhy a Modular ABC ApproachModular ABC Approach

Many new products, features,technologies

HA and HP application operation isthe goal

Start with modular, structuredapproach (the logical design)

Add multicast, VoIP, DPT, DWDM...



Price per 10/100

Catalyst 2912GCatalyst 2948GCatalyst 2980G

242410/100 Ports10/100 Ports

Gigabit PortsGigabit Ports

24-500+24-500+ 24-350+24-350+

3-38+3-38+ 8-64+8-64+

Catalyst 5XXX

32-9632-96

6-126-12

Catalyst 4XXX

$100

$200

$250

$300

$350

Switching CapacitySwitching Capacity Up to 72 MppsUp to 72 Mpps20 Mpps20 Mpps Up to 150 MppsUp to 150 MppsBackplaneBackplane 24 Gbps24 Gbps 1.2-3.6 + 10Gbps1.2-3.6 + 10Gbps 250+ Gbps250+ Gbps

NewNew

NewNew

NewNew

ModulesModules

Catalyst 6XXX

Design the SolutionDesign the SolutionThen Pick the ProductsThen Pick the Products

NewNewModulesModules


9/589


HA Design Reality Check!HA Design Reality Check!Assume Things FailAssume Things Fail -- Then What?Then What?

Networks are complex Things break, people make mistakes

What happens if a failure occurs? Simple, structured, deterministic design

required for fast recovery The tradeoffs

your choices are important


10/5810


Layer 2Layer 2

Layer 2Layer 2AccessAccess

DistributionDistribution

BuildingBuilding

Core L3Core L3

ServerServerDistributionDistribution

Server FarmServer Farm

Layer 3Layer 3

3

21

5

6

BranchesBranches

WAN

WANbackup

4

Network RecoveryNetwork RecoveryHow Long? What Happens?How Long? What Happens?


11/5811


FailureScenarioFailure

Scenario

1,2 server

3,4 uplink

5,6 core

dual-path L3

EtherChannel

L3 routing

L2 general

DPT

1,2 server

3,4 uplink

5,6 core

dual-path L3

EtherChannel

L3 routing

L2 general

DPT

RecoveryMode

RecoveryMode

RecoveryTime

RecoveryTime

Server NIC

HSRP (& UplinkFast)

HSRP track

alternate path used

channel recovery

EIGRP or OSPF

L2 spanning tree

IPS

Server NIC

HSRP (& UplinkFast)

HSRP track

alternate path used

channel recovery

EIGRP or OSPF

L2 spanning tree

IPS

< 2 seconds

tune to 3 seconds

tune to 3 seconds

< 2 seconds

< 1 second

depends on tuning

tune (up to 50 seconds)

50 milliseconds

< 2 seconds

tune to 3 seconds

tune to 3 seconds

< 2 seconds

< 1 second

depends on tuning

tune (up to 50 seconds)

50 milliseconds

Network Recovery TimesNetwork Recovery TimesIf You Follow the RulesIf You Follow the Rules


12/58


Design for High AvailabilityDesign for High AvailabilityHow to Build Boring Networks!How to Build Boring Networks!

The Concepts The Rules Design Building Block

Design Backbone Notes on Tuning


13/58


HA Network Design ConceptsHA Network Design Conceptsthinking outside the boxthinking outside the box

1) Simplicity & Determinism2) Collapse the Sandwich3) Spanning Tree Failure Domain4) Map L3 to L2 to L15) Scaling and Hierarchy6) ABCs of Module + Backbone

Design

7) The Four Corners


14/58


1) Simplicity and Determinism1) Simplicity and Determinismreducing the degrees of freedomreducing the degrees of freedom

Every Choice Affects Availability! Determinism or Flexibility?

Would you support 27 desktop environments? Would you support 13 network vendors? Would you use 57 varieties of Cisco IOS?

FlexibleComplex

Varied

SimpleStructured

DeterministicHA Continuum

Boring! Interesting!


15/58


TraditionalModel

Fiber

SONET

Big Fat Pipe

Lower equipment cost

Lower operational cost

Simplified architecture

Scalable capacity

OpticalInternetworking

Fiber

IP

FR/ATM

IP

2)2) Collapse the SandwichCollapse the Sandwichroute IP over glassroute IP over glass

Service

TrafficEng

Fiber

Mgmt

33)) Minimize the Failure DomainMinimize the Failure Domain


16/58


33)) Minimize the Failure DomainMinimize the Failure Domainpublic enemy number onepublic enemy number one

Where should root go?

What happens whensomething breaks?

How long to converge?

Many blocking links

Large failure domain!

Broadcast flooding

Multicast flooding

Loops within loops

ST from heck

Times 100 VLANs?

avoid highly meshed, non-deterministic large scale L2 = VLAN topology

Building 1 Building 2

Building 3 Building 4


17/58


4)4) Map L3 to L2 to L1Map L3 to L2 to L1

Easier administration & troubleshooting

Clients in subnet 10.0.55.0

VLAN 55

wiring closet 55 on floor 55

access switch 55

interface VLAN 55

all match and life is good

go fishing with your kids

10/100 BaseT

GE or GEC


18/58


5) Scaling and Hierarchy5) Scaling and Hierarchy

Strong hierarchieslike telephonesystem andInternet segmentaddressing andtherefore scale

U

C

N

U

C

N

U

C

N

C complexityU unmanageableN number of devices

Flat L2 Ethernet iseasy but does notscale

ATM LANE islogically flat, scalesas N squared

6)6) B ilding Block &Building Block &


19/58


6)6) Building Block &Building Block &Backbone Design ABCsBackbone Design ABCs

WAN

EcommerceSolution

PSTN


CoreCore

LAN AccessLAN Access


Server Farm

Internet

A design bb

B design BB

C connect bb to BBDivide and conquer

Cookie cutterconfiguration

Deterministic

L3 demarcation

WAN AccessWAN Access

) k d d


20/58


7) Four Square Network Redundancy7) Four Square Network Redundancyor the Four Corners Problemor the Four Corners Problem

One ChassisOne Chassis Two ChassisTwo Chassis

OneOneSupervisorSupervisor

TwoTwoSupervisorsSupervisors

SimplestSimplestNo RedundancyNo Redundancy

Most ComplexMost Complex

Belt and SuspendersBelt and Suspenders

GeoPhysicalGeoPhysicalEffectiveEffective

When spaceWhen space

is limitedis limited

HAHA

L3L3


21/58


Dos and Donts for HA DesignDos and Donts for HA Design

1) Eliminate STP Loops2) L3 Dual-Path Design3) EtherChannel Across Cards4) Workgroup Servers

5) Use HSRP Track6) Passive Interfaces7) Issues with Single-Path Design

8) Oversubscription Guidelines9) HA for single attached servers10) Protocol Tradeoffs

11) UDLD Protection

l ) lR l 1) Eli i STP L


22/58


Rule 1) Eliminate STP LoopsRule 1) Eliminate STP Loopsin the backbone and mission critical pointsin the backbone and mission critical points

No blocking links towaste bandwidth

Avoids slow STPconvergence

Very deterministic

Routed links not VLANtrunks

L2 Gigabit switch inbackbone

subnet X = VLAN X

Too many cooks spoil the broth

L3 control is better

X.2 X.3X.1

RootVLAN X


23/58


Rule 2) Dual EqualRule 2) Dual Equal --Cost Path L3Cost Path L3

Load balance - dont waste bandwidthunlike L1 and L2 redundancy

Fast recovery to remaining pathdetect L1 down & purge - about 1s

Works with any routed fat pipes

Path A

Path B

Destinationnetwork X

Equal costroutes to XPath A

Path B


24/58


Rule 3)Rule 3) EtherChannelEtherChannel Across CardsAcross Cards

Increased availability Sub second recovery Spans cards on 6500 Up to 8 ports in channel

Small complexity increase Single L2 STP link Single L3 subnet less if channel set on


25/58


Rule 4a) Connect Workgroup ServerRule 4a) Connect Workgroup Server

With no L2 recovery path, what happens if linkbreaks .

Workgroup server X.100attached to distribution layer

L2 path to client X.1

Client X.1 VLAN X in purple

includes clientsand workgroupservers attachedat different places.

A B

C

Links to core

Link CBbreaks .


26/58


Rule 4b) Connect Workgroup ServerRule 4b) Connect Workgroup Server

Subnet X now discontiguous Incoming traffic gets dropped

Workgroup server X.100attached to distribution layer

L2 path to client X.1

Client X.1

Routers A & B continue toadvertise reachability ofsubnet X ...

A B

C

X.1 not

reachable

X.100 not

reachable


27/58


Rule 4c) Connect Workgroup ServerRule 4c) Connect Workgroup Server

Introduce L2/STP redundancy Adds a loop (band-aid fix)

Workgroup server X.100

attached to distribution layerL2 path to client X.1

Client X.1VLAN trunk AB forms L2 looprecovery path for STP

prevents black hole

A B

C


28/58


Rule 4d) Connect Workgroup ServerRule 4d) Connect Workgroup Server

Real Lessons: Enterprise Server Farmsare better L3 demarcation is better Example of why extendedL2 is difficult


29/58


Rule 5a) Use HSRP TrackRule 5a) Use HSRP Track Review - Hot Standby Router Protocol

Fast recovery can be tuned to 3s or less

X is M.100HSRP PrimaryPriority 200

Y ( becomes M.100)HSRP BackupPriority 100

Z

Router X acts as gatewayrouter for subnet M, IP addressM.100. If link Z fails router Y

will take over as M.100 gatewaywith same MAC address

10/100 BaseT

GE or GEC

Subnet Mhosts M.1 M.2 M.3


30/58


Rule 5b) Use HSRP TrackRule 5b) Use HSRP Track

Track extends HSRP to monitor links to backbone

Ensures shortest path - best outbound gateway

Track interface A - lower priority 75Track interface B - lower priority 75HSRP triggers if both A and B lost

10/100 BaseT

GE or GEC

X is M.100HSRP PrimaryPriority 200

Y ( becomes M.100)HSRP BackupPriority 100

Z

Subnet Mhosts M.1 M.2 M.3

A B

l ) f


31/58


Rule 6a) Use Passive InterfacesRule 6a) Use Passive Interfaces

L3 switches X & Y in distribution layer 4 VLANs per wiring closet

10 wiring closets

X Y

ABCD EFGH IJKL MNOP

Ten totalWiringcloset

switch

Distributionswitch

l b) fR l 6b) U P i I f


32/58


Rule 6b) Use Passive InterfacesRule 6b) Use Passive Interfaces

What X and Y see is 4*10=40 routed links Increased protocol overhead & CPU

X Y

A

CB

DEFG

Etc.

A.1

C.1B.1

D.1E.1F.1G.1

Etc.

A.2

C.2B.2

D.2E.2F.2G.2

Etc.

R l 6 ) U P i I fR l 6 ) U P i I f


33/58


Rule 6c) Use Passive InterfacesRule 6c) Use Passive Interfaces

Turns off routing updates & overhead

Leave two routed links for redundant paths CDP, VTP, HSRP etc. still function on all links

X Y

A

CB

DEFG

Etc.

A.1

C.1 (passive)B.1 (passive)

D.1 (passive)E.1F.1 (passive)G.1 (passive)

Etc.

A.2

C.2 (passive)B.2 (passive)

D.2 (passive)E.2F.2 (passive)G.2 (passive)

Etc.

Rule 7a) Issues With Single PathRule 7a) Issues With Single Path


34/58


Rule 7a) Issues With Single PathRule 7a) Issues With Single PathDesignsDesigns

L3 engine MSFC oncore-X reloads Lights are on but

nobody home - HSRPdoes not recover Remove passive

interface to wiringcloset subnets A, B

Provide longer routedrecovery path

Single path

to core

GE

Subnet A Subnet B

X

HSRPprimary

Core L3

Access

L2

Y

New, longeroutboundroutes

Outbound case ...

Rule 7b) Issues with SingleRule 7b) Issues with Single --PathPath


35/58


Rule 7b) Issues with Single) g PathDesignDesign

Recovery must takeplace in bothdirections

Routing protocolrecovers longer routefrom X to subnets A, B

Therefore dual-path L3is better & faster thansingle-path

Single path

to core

GE

Subnet A Subnet B

X

HSRPprimary

Core L3

Access

L2

Y

New, longerroutes to A, B

Inbound case ...

Rule 8a)Rule 8a) OversubscriptionOversubscription


36/58


)) ppGuidelinesGuidelines

Oversubscription part ofall networks - not bad

Non-blocking switchesdo not mean a non-blocking network

You determine theamount of blocking

GE

GE

Non-blockingdesign

GE

GE

Blockingdesign 2:1

GE

Rule 8b)Rule 8b) OversubscriptionOversubscription


37/58


)) ppGuidelinesGuidelines

Oversubscription rulesof thumb work well

20:1 at wiring closet

Less in distribution andserver farm QoS required IFF

congestion occurs Protect real time flows

at congested points

n:1

20:1

Core L3use non-blockingswitches

Dual-linkGEC

200 100BaseT

GE8 uplinks

DistributionL3

Rule 9) Dual SupervisorsRule 9) Dual Supervisors


38/58


) p) pHA for Single Attached ServersHA for Single Attached Servers

Single point of failure Dual supervisors - fast stateful recovery No increase in complexity

10/100 BaseT

GE or GEC

Single attached servermission critical application

HA dual supervisorsCatalyst 6XXX

Redundant uplinks

Rule 10)Rule 10) Protocol TradeoffsProtocol Tradeoffs


39/58


))Automatic or Manual ConfigurationAutomatic or Manual Configuration

Configuration up front rather

than CPU overhead later, forexample: set VTP mode transparent set/clear VLANs for each trunk set trunks on or off

set channel on or off Choose flexibility or

determinism

Rule 11)Rule 11) UniDirectionalUniDirectional LinkLink


40/58


)DetectionDetection

UDLD detects mismatch when physical layerchecks out OK

Prevents various failure conditions includingcrossed wiring

Tx Fiber

Rx Fiber

The lights

are on,BUT ..

Building Block Means SurvivableBuilding Block Means Survivable


41/58


ggSelfSelf --contained Backbonecontained Backbone

Autonomous SurvivabilityUnit - HSRP

L3 Broadcast Multicastdemarcation

Cookie cutter configuration L3 Demarcation of failure

domain

Simple, repeatable,deterministic Redundancy adds 15% cost

at mission critical points likeserver farm

L2L3

ASUdelimitsfailuredomain

Building Block TemplatesBuilding Block Templates


42/58


g pUse As Is or CombineUse As Is or Combine

1) Standard Model

simple, structured2) VLAN Model

more flexible3) Large Scale Server Farm

Modelaccommodate dual NIC

4) Small Scale Server FarmModel

accommodate dual NIC

1) Standard Building Block1) Standard Building Block


43/58


no loopsno loops -- no STP complexityno STP complexity

HSRP PrimarySubnets/VLANs10, 12, 14, 16

HSRP PrimarySubnets/VLANs11, 13, 15, 17

Access L2root switch

VLAN 10/11

Subnet 10Subnet 11

GE/GECVLAN Trunks

10/100 BaseT

GE or GEC Dual Path with Tracking

Subnet 12Subnet 13

Subnet 14Subnet 15

Subnet 16Subnet 17

Highly DeterministicL1 maps L2 maps L3

No blocking linksShortest path alwaysNot flexible

2) VLAN Building Block2 u ng ocmake L2 design match L3 designmake L2 design match L3 design


44/58


AllAll VLANsVLANs terminate at L3 boundaryterminate at L3 boundary

STP rootVLANs 10 12 14 16

HSRP primarysubnets 10 12 14 16

STP rootVLANs 11 13 15 17

HSRP primarysubnets 11 13 15 17

L2L3

All VLANsAll Subnets

GE/GECVLAN Trunks

Dual Path with Tracking




L2Path

10/100 BaseT

GE or GEC

More flexibleFO forwarding oddBE blocking even etc.

FEBO

FOBE

FEBO

FOBE

FEBO

FOBE

FEBO

FOBE

L2L3

Uplink-

Fast

3) Large3) Large --Scale Server FarmScale Server Farm


45/58


Building BlockBuilding BlockDual-NIC ServerExample Fault Tolerant Mode (FTM)Same IP Address - seamless recovery

GE/GECVLAN Trunks


L2Path

Access L2UplinkFast

10/100 BaseT

GE or GEC

based on VLAN building blockaggregates traffic - high BW

L2L3

L2L3

STP rootVLANs EVEN

HSRP primarysubnets EVEN

STP rootVLANs ODD

HSRP primarysubnets ODD

4) Small4) Small --Scale Server FarmScale Server Farm


46/58


Building BlockBuilding Block

Dual-NIC ServerExample Fault Tolerant Mode (FTM)

Same IP Address - seamless recovery


L2Path

10/100 BaseT

GE or GEC

Simplified building block withno STP loops

Use if port density permits

Use if no oversubscription(non-blocking) is arequirement

L2L3

L2L3 HSRP primarysubnets EVEN HSRP primarysubnets ODD

Redundant Backbone ModelsRedundant Backbone Models


47/58


all goodall good -- increasing scaleincreasing scale

1) Collapsed L3 Backbone2) Full Mesh

3) Partial Mesh

4) Dual-Path L2 Switched

5) Dual-Path L3 Switched

1) Collapsed L3 Backbonelarge building or small campus


48/58


Core L3

Access L2

large building or small campus

Clients

CollapsedBackbone

GE/GECScale depends onphysical plant and

policy more thanperformance

Server Farm10/100 BaseT

GE or GEC

2) Full Mesh Backbonesmall campus - n squared limitation


49/58


Client

Blocks Distribution L3

Access L2

ServerBlock

Distribution L3

Access L2

Note importance ofpassive wiringcloset interfaces inmeshed designs!

2 blocks - 6 peerings3 blocks - 15 peerings4 blocks - 28 peerings5 blocks - 45 peerings

E or FE PortGE or GEC

3) Partial Mesh Backbone3) Partial Mesh Backbonemedium campusmedium campus -- traffic flow to server farmtraffic flow to server farm


50/58


Distribution/Core L3

Access L2

ClientBlocks Distribution L3

Access L2

medium campusmedium campus -- traffic flow to server farmtraffic flow to server farm

ServerBlock


Predominanttraffic pattern

4) Dual4) Dual --Path L2 Switched BackbonePath L2 Switched Backboneno STP loops or VLAN tr nks in coreno STP loops or VLAN trunks in core


51/58


no STP loops or VLAN trunks in coreno STP loops or VLAN trunks in core

South

ClientBlocks

Dual L2 Backbone

Distribution L3

Core L2

Access L2

red coresubnet=VLAN=ELAN

blue coresubnet=VLAN=ELAN

WestNorth


5a) Benefits of a L3 Backbone5a) Benefits of a L3 Backbone


52/58


))

Multicast PIM routing control Load balancing No blocked links Fast convergence EIGRP/OSPF Greater scalability overall Router peering reduced IOS features in the backbone

5b) Dual-Path L3 Backbonelargest scale, intelligent multicast


53/58


Distribution L3

Access L2

g , g

Core L3

ServerFarmBlock

Distribution L3

Access L2

All routed links,consider subnetcount !

ClientBlock


Restore ConsiderationsRestore Considerations


54/58


Restoring can take longer insome cases - more complex -schedule

On power up L1 may come upbefore L3 builds routing table -temporary black hole for HSRP

Use preempt delay for HSRP

Restoring can take longer insome cases - more complex -schedule

On power up L1 may come upbefore L3 builds routing table -

temporary black hole for HSRP Use preempt delay for HSRP

Campus Failover Layer 2Campus Failover Layer 2Recovery & TuningRecovery & Tuning


55/58


Recovery & TuningRecovery & Tuning

STPTune diameter onroot switchImproves recoverytime maxage

PortFastServer or desktopports only 1 s

Move directly fromlinkup into

forwarding

UplinkFastNo tuning, 2seconds, wiringcloset onlyOnly applies withforwarding &blocking link

BackbonefastConverges 2 sec +2xFwd_delay forindirect link failures

Eliminates maxage

timeout

Campus Failover Layer 3Campus Failover Layer 3Recovery & TuningRecovery & Tuning


56/58


Recovery & TuningRecovery & Tuning

Caution withaggressive tuning

Good when networkis stable, highlysummarized

HSRP (fast LAN links)Tune hello timer 1sec, dead timer 3 sec


57/58


KISS - eliminate complex L2 ASU - building blocks Redundant backbone

Redundant L3 paths L3 segments failure domain


58/58


cisco hight availability enterprise network design

Documents