ospf to isisvijay/work/ppt/oi.pdf · 2003. 10. 26. · vijay gill jon mitchell [email protected]...
TRANSCRIPT
![Page 2: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/2.jpg)
Notes
"But in our enthusiasm, we could notresist a radical overhaul of thesystem, in which all of its majorweaknesses have been exposed,analyzed, and replaced with newweaknesses."
-Bruce Leverett
![Page 3: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/3.jpg)
Why
• Features– Convergence
• Security
• Simplicitybut to learn ISIS you have to know the secret
handshake and be a *%##%ing 33rd levelmason
-Chance Whaley
![Page 4: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/4.jpg)
Security
• http://www.nanog.org/mtg-0006/katz.html
• Packet bombs
• Wasn’t as big of a deal for AOL– We have packet filters on most line cards
• Most is not ALL
• Runs directly on L2– Harder to spoof or attack
![Page 5: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/5.jpg)
Simplicity
• Found out we didn’t need areas– Added complexity
• Configuration
• Typos
• Slowed it down– DV
– Flat area easy to configure and maintain• Stupid, but no stupider (apologies to Einstein)
![Page 6: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/6.jpg)
State of the Art
POP3
BB1 BB2
L0: x.y.z.n
N.N
.N.m
/31 N.N .N.o/31
OSPF AREA 0
OSPF AREA X
ATDN OSPF
POP1
P6/0
P4/0/0
P6/1
P5/0/0
P2/0P2/0
POP2
P1/0P0/0 P1/0P0/0
P0/0 P1/0 P6/0P0/0 P1/0
L0: x.y.z.m
L0: A.B.C.DBlock: X.Y.A.B/28
P6/2to
bb2-ZZZ
to bb2 -XX X
BB Sample Config
router ospf 1log-adjacency-changesarea 0 authenticationarea x authentication
passive-interface Loopback0network A.B.C.0 0.0.3.255 area 0network A.B.D.0 0.0.1.255 area Xnetwork A.B.C.0 0.0.7.255 area X
maximum-paths 6
area X range A.B.C.x 255.255.255.240area X range A.B.C.y 255.255.255.240
Note: Area X is the BGP cluster-ID ofthe site
POP Sample Config
router ospf 1log-adjacency-changesarea X authenticationredistribute connected subnetspassive-interface Loopback0network A.B.C.0 0.0.1.255 area X….Maximum-paths 6
X. Y.A.B/31
N.N.M.Y/31P6/2
A.B.C
.E/3
1to p
op1-YYY
P4/0
A. B.C
.D/30
L0: A.B.C.EBlock: X.Y.A.C/28
L0: A.B.C.FBlock: X.Y.A.D/28
![Page 7: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/7.jpg)
Strategy
Ships in night– Run parallel– Verify routes
• Raise OSPF admin distance• Verify network after change• Remove OSPF
The plan is in the works, but we have not activated the implementation phase.-Frank Caddeo
![Page 8: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/8.jpg)
Main Backbone Nodes
bb2-mtc
bb2-dtc
bb2-dcl
bb2-nyc
bb1-dtc
bb1-frabb1-nye
bb2-spo
bb1-nyc
bb1-mtc
bb1-dcl
bb1-frr
bb2-frabb2-nye
bb1-spo
bb2-frr
bb1-loh
bb2-loh
bb1-tkn
bb2-sun
America OnlineInternet Operations
12100 Sunrise Valley Drive, Reston, VA 20191
Date:
Revision:
Drawn:
October 17, 2003
6.2
tdo
Architect:
AOL Proprietary and Confidential
Updated: tdo
bb2-sje
bb2-ash
bb1-ash
bb1-den bb2-den
bb1-new
bb2-new
bb1-alb
bb2-alb
bb1-hon
bb2-hon
bb2-seabb1-sea
bb1-kcybb2-kcy bb1-ch1
bb2-chi
bb1-sun
bb2-ntc
bb1-ntc
bb1-sje
bb2-tkn
bb2-las
bb1-las
bb2-phobb1-pho
bb1-col
bb2-col
bb1-hou
bb2-hou
bb1-tbybb2-tby
bb1-atm
bb2-atm
bb2-cha
bb1-cha
bb2-vie
bb1-vie
bb2-rtc
bb1-rtc
2x 48
2x48
2x 48
2x48
bb1-rtl
bb2-rtl
bb1-prs
bb2-prs
bb1-cin bb2-cin
bb1-sjg
bb2-sjg
bb2-dls
bb1-dls
![Page 9: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/9.jpg)
Out of Band
“OOB is the saving throw when you @#$%up”
-RS
• Verified OOB reachability to all POPsbeforehand
![Page 10: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/10.jpg)
IS-IS Migration Prep
Pre-Migration– Load IS-IS configuration built with scripts on RTL
routers• Non Customer PoP
– Develop/test scripts to check IS-IS neighborrelationships and route consistency
![Page 11: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/11.jpg)
Migration Week
• Load IS-IS configuration
• Verify IS-IS neighbor relationships
• Verify LSPs in IS-IS database
• Change OSPF administrative distance to 254
– On some edge routers
Some mornings, it's just not worth chewing through the leather straps.-Emo Phillips
![Page 12: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/12.jpg)
Migration Week (cont)
• Compare IS-IS and OSPF routes on pair of pop routers
• IS-IS vs. OSPF cost check on all interfaces in network
• Change OSPF administrative distance of all remainingrouters to 254
• Verify no OSPF routes in fowarding table
• Basic network reachability
– Ping all routers
– Check connectivity to some external sites• Standard NOC monitoring
Your rules are really beginning to annoy me-Snake Plissken
![Page 13: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/13.jpg)
Post-Migration
• No verification– Verification done as part of migration
• Run a script to remove the OSPF configurationfrom all ATDN routers
We had more than enough genuine headaches as it was, and trivial aestheticconcerns weren't even close to making it onto our agenda.
-Geoff Miller
![Page 14: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/14.jpg)
Current Setup
S1/0/0:0 S1/1/0:0
POP2
BB1 BB2
To bb2-den P7/0
to bb1-chi p6/0
POP1 POP3
P0/0
P0/0
P0/0P0/1 P0/1
P1/0P1/0
P1/0P1/0P7/0 P7/0
P0/0
P8/0P8/0
P5/0/0 P8/0/0
to bb2-dal P6/0
P6/0
CustomerAS: Blah
Low Speed CustomerAS: Blah
P3/0P3/0 P3/1 P3/1
PeerAS: Blah
P0/2
503 503 503 505 503 505
1
1010
10
# IS-ISMETRIC
OC-192
OC-48
OC-12
OC-3
DS1
GSR 12410GSR 12410
GSR 12410 GSR 12410 7513
![Page 15: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/15.jpg)
Config Bits• !• interface Loopback0• isis metric 1 level-2• !• interface POS5/0• description P5/0: bb1-nye-P5-0-pop1-nye-P5-0 (66.p.x.y/31 direct-cabled)(T=pbNYE)• ip router isis• isis metric 503 level-2• isis password ISISPASSWORD(hint, this isn’t the real password) level-2• !• router isis• passive-interface Loopback0• maximum-paths 6• net 39.752f.0100.0014.0000.5000.1668.router.id.inIPv4.00
• is-type level-2-only !Why Level 2?• domain-password this-isn’t-the-real-password-either
• metric-style wide !• external overload signalling ! Ensure that IS-IS will tear down
adjacencies when dCEF is disabled on an interface
• set-overload-bit on-startup wait-for-bgp ! Avoid placingrouter on IGP SPF before bgp
• max-lsp-lifetime 65535• lsp-refresh-interval 65000
• no hello padding ! Hello padding to mtu is deprecated• log-adjacency-changes all• !
![Page 16: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/16.jpg)
Design
• All connected interfaces are redistributed intoBGP
• IS-IS will be preferred
• Redistribution into BGP chosen to reduce thenumber of links in the SPF– Is it an issue in practice
• Not really
cluelessness leads to flapping... flapping leads todampening... dampening leads to suffering
-RS
![Page 17: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/17.jpg)
Cost and RR Design
• Backbone links– Used OSPF metrics– BB-POP Interconnects
• OSPF metric + 500• Avoids Inversion on BB-BB link failure• Mirrors OSPF w/ Areas behavior
– MED oscillation issue• Full mesh of POP routers• No client-to-client reflection• Cost (InterPOP) > cost differences IntraPOP
• New cost out procedure– add 10000 to the interface
![Page 18: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/18.jpg)
Timeline
LoadISIS config
Day
Tim
e
Verify routes
SwitchDistance
ConfirmReach
RemoveOSPF
![Page 19: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/19.jpg)
Loading ISIS Config
• Non Disruptive
• Config was loaded in a three hour window,Monday 6-9 am
• Script (OSPF) -> IS-IS
• Output was copied to each router
• No IS-IS routes in use
If you can't remember, then the claymore is pointed at you
![Page 20: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/20.jpg)
Route Verification
• Compare IS-IS neighbor topology with OSPF– show clns neighbor
– show ip ospf neighbor
• Check IS-IS database on all routers– Ensure all other routers LSP’s installed in IS-IS
database (sh isis database)
![Page 21: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/21.jpg)
Route Verification
• On selected edge routers– Change OSPF admin distance to 254
– Verify traffic to peers
– Compare IS-IS and the OSPF routes• All routes in the network are correctly in IS-IS?
• Go or No Go
Great ideas, in theory, should not be hampered bypesky reality
-Dys
![Page 22: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/22.jpg)
MED
• Changing metric affects MEDs– New metric in the BGP one minute after distance
change– Ratchet down
• Does not propagate for another 10 minutes
– One Large Peer – LP• Listened to MEDs• Not enough capacity to fit all of traffic in one circuit
– All routers connected to LP• Migrated at roughly the same time• Manually cleared soft out after the metric advertisement
updated
![Page 23: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/23.jpg)
The Big One
• Flip Admin Distance– IS-IS routes are preferred
• Current network metrics are consistent with config files?
• Slow Start– Manually change admin distance to 254 on more edge POPS
• Go No-Go?
• Script to flip the rest– From the edge to the center (with respect to ops2)
– In order - LP, europe, asia, brazil, us-pop, us-bb, and dc
• External routes in OSPF now in iBGP
![Page 24: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/24.jpg)
Routing
• Convergence time for the installation– <1 second
• No CEF updates– Costs changed but PATHS didn’t
• All production traffic is routed to Edgerouter loopbacks (n-h-s)
• Rollback• Remove admin distance command• Pre-written script
This thing severely violated the Rule of Complexity as applied to the problem. The Rule ofComplexity states that if an answer seems too complicated to be the right answer, it is the
wrong answer.-Steve Cutchen
![Page 25: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/25.jpg)
Removal of OSPF configuration
• After burn in– 0300 EDT
– OSPF configuration removed• Non-disruptive change
– Old OSPF configs archived via RANCID
We are jolly green giants, walking the earth with routers.-Christopher Morgan (after no router ospf 10 at MFN)
![Page 26: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/26.jpg)
Subject: From the install fileDate: 6/25/2003To: [email protected]: John
Network Install Doc for Non-Bounce June 25, 2003General Maintenance (times noted with attribution):
c) Switching ATDN backbone from OSPF to ISIS as the igp. 0300 Expected Impact: None
Dog will hunt/vijay
Line of Truth
![Page 27: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/27.jpg)
Traffic
![Page 28: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/28.jpg)
![Page 29: OSPF to ISISvijay/work/ppt/oi.pdf · 2003. 10. 26. · Vijay Gill Jon Mitchell jrmitche@aol.net vijaygill9@aol.com. Notes "But in our enthusiasm, we could not resist a radical overhaul](https://reader033.vdocuments.site/reader033/viewer/2022060519/604c96708ad44240eb2ebb42/html5/thumbnails/29.jpg)
Questions?
You thinking about smoking off the MPLS hookah?-Brook Bailey
There is a difference between making something foolproof andreducing the number of fools
-Bill Barns