isp network design - network startup resource center … · isp network design isp/ixp workshops...
TRANSCRIPT
1
1© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Network DesignISP/IXP WorkshopsISP/IXP Workshops
222© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Network Design
• PoP Topologies and Design• Backbone Design
• ISP Systems Design
• Addressing• Routing Protocols
• Security
• Out of Band Management• Operational Considerations
3© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Point of Presence Topologies
444© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Topologies
• Core routers – high speed trunk connections• Distribution routers and Access routers – high
port density
• Border routers – connections to other providers• Service routers – hosting and servers• Some functions might be handled by a single
router
555© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Design
• Modular Design• Aggregation Services separated according to
connection speedcustomer servicecontention ratiosecurity considerations
666© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Modular PoP Design
Backbone linkto another PoP
Backbone linkto another PoP
Nx64 customeraggregation layer
Nx64 leased line circuit deliveryChannelised T1/E1 circuits
Hosted ServicesISP Services(DNS, Mail, News,
FTP, WWW)
NetworkOperations
Centre
ConsumerDIAL Access
Other ISPsWeb Cache
NetworkCore
Consumer cable, xDSL and
wireless Access
NxT1/E1 customeraggregation layer
T1/E1 leased line circuit deliveryChannelised T3/E3 circuits
2
777© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Modular Routing Protocol Design
• Modular IGP implementationIGP “area” per moduleaggregation/summarisation where possible into the core
• Modular iBGP implementationBGP route reflector cluster per modulecore routers are route-reflectorsclients peer with core only
8© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Point of Presence Design
999© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Modules
• Low Speed customer connectionsPSTN/ISDN dialuplow bandwidth needslow revenue, large numbers
• Medium Speed customer connections56/64K to sub-T1/E1 speedslow bandwidth needsmedium revenue, medium numbers
101010© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Modules
• High Speed customer connectionsE1++ speedsmedium bandwidth needshigh revenue, low numbers
• Broad Band customer connectionsxDSL, Cable and Wirelesshigh bandwidth needslow revenue, large numbers
111111© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Modules
• PoP CoreTwo dedicated routersHigh Speed interconnectBackbone Links ONLYDo not touch them!
• Border Networkdedicated border router to other ISPsthe ISP’s “front” doortransparent web caching
121212© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Modules
• ISP ServicesDNS (cache, secondary)News, Mail (POP3, Relay)WWW (server, proxy, cache)
• Hosted ServicesVirtual Web, WWW (server, proxy, cache)Information/Content ServicesElectronic Commerce
3
131313© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
PoP Modules
• Network Operations Centreprimary and backup locationsnetwork monitoring
statistics and log gatheringdirect but secure access
• Out of Band Management NetworkThe ISP Network “Safety Belt”
141414© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Low Speed Access Module
To Core Routers
Primary Rate T1/E1
PSTN lines tomodem bank
PSTN lines tobuilt-in modems
AS5300
AS2511
2600/3600
TACACS+/Radiusproxy, DNS resolver,
Content
Web Cache
Access NetworkGateway Routers
151515© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Medium Speed Access Module
To Core Routers
Channelised T1/E1
64K and nx64K circuits
Mixture of channelisedT1/E1, 56/64K and
nx64K circuits
3800/7206/7600
161616© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
High Speed Access Module
To Core Routers
Channelised T3/E3
T1 and E1 circuits
Mixture of channelisedT3/E3 and T1/E1 circuits
7200/7600
171717© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Broad Band Access Module
To Core Routers
Telephone Network
The cable system
6400
SSG, DHCP, TACACS+or Radius Servers/Proxies,
DNS resolver, Content
Web Cache
Access NetworkGateway Routers
uBR7246
61xx
IP, ATM
181818© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services Module
DNScache
DNSsecondary POP3 Mail
Relay NEWS
To core routers
WWWcache
Service NetworkGateway Routers
4
191919© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Hosted Services Module
Customer 7Customer 3Customer 4
Customer 5Customer 6
To core routers
Hosted NetworkGateway Routers
Customer 2Customer 1
202020© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Border Module
To core routers
NetworkBorder Routers
To local IXP -NB - no default route +
local AS routing table only
ISP1 ISP2
212121© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
NOC Module
Primary DNS
To core routers
Hosted NetworkGateway Routers
SYSLOGserver
TACACS+server
Network Operations Centre Staff
Out of BandManagement Network
2620/32async
NetFlowAnalyser
Firewall
Billing, Database and Accounting
Systems
Corporate LAN
Critical ServicesModule
222222© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Out of Band Network
Out of BandManagement Network
2620/32asyncTo the NOC
Out of Band Ethernet
NetFlowCollector
NetFlowenabledrouters
Routerconsoles
23© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Backbone Network Design
242424© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Backbone Design
• Routed Backbone• Switched Backbone• Leased point-to-point circuits
nx64K, T1/E1, T3/E3, OC3, OC12,...
• ATM/Frame Relay service from telcoT3, OC3, OC12,… deliveryeasily upgradeable bandwidth (CIR)
5
252525© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Distributed Network Design
• PoP design “standardised”operational scalability and simplicity
• ISP essential services distributed aroundbackbone
• NOC and “backup” NOC
• Redundant backbone links
262626© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Distributed Network Design
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections Operations Centre
BackupOperations Centre
ISP Services
ISP Services
ISP Services
272727© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Backbone Links
• ATM/Frame Relaynow less popular due to overhead, extra equipment,and shared with other customers of the telco
• Leased Linemore popular with backbone providers
IP over Optics and MPLS coming into the mainstream
282828© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Long Distance Backbone Links
• Tend to cost more• Plan for the future (at least two years ahead)
but stay in budgetUnplanned “emergency” upgrades can be disruptivewithout redundancy
• Allow sufficient capacity on alternative pathsfor failure situations
sufficient can be 20% to 50%
292929© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Long Distance Links
POP One
POP Two
POP Three
Long distance link
Alternative/Backup Path
303030© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Metropolitan Area Backbone Links
• Tend to be cheaperCircuit concentrationChoose from multiple suppliers
• Think bigMore redundancyLess impact of upgradesLess impact of failures
6
313131© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Metropolitan Area Backbone Links
POP One
POP Two
POP Three
Metropolitan Links
Metropolitan Links
Traditional Point to Point Links32© 2005, Cisco Systems, Inc. All rights reserved.
Cisco ISPWorkshops
ISP Services
DNS, Mail, Newsdesign and location
333333© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• Domain Name SystemProvides name and address resolutionServers need to be differentiated, properlylocated and specified
Primary nameserverSecondary nameserverCaching nameserver – resolver
343434© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• Primary nameserverHolds ISP zone files
forward zone (list of name to address mappings) for allISP’s and any customer zonesreverse zone (list of address to name mappings) for allISP’s address space
One Unix server, fast I/O, reasonable amount ofmemory (512Mbytes), reasonable diskLocated in secure part of net, e.g. NOC LAN
353535© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• Secondary nameserverHolds copies of ISP zone filesAt least two are required, more is betterUnix server, fast I/O, reasonable amount of memory(512Mbytes), reasonable diskShould be geographically separate from each otherand the primary DNS
At different PoPsOn a different continent e.g. www.secondary.comAt another ISP
363636© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Secondary DNS Example
• apnic.net zoneprimary DNS in Brisbanesecondary DNS around the world
$ dig apnic.net ns
;; ANSWER SECTION:apnic.net. 50m44s IN NS svc00.apnic.net.apnic.net. 50m44s IN NS ns.ripe.net.apnic.net. 50m44s IN NS rs.arin.net.apnic.net. 50m44s IN NS ns.apnic.net.
;; ADDITIONAL SECTION:svc00.apnic.net. 1d23h53m25s IN A 202.12.28.131ns.ripe.net. 1d23h54m46s IN A 193.0.0.193rs.arin.net. 1d23h53m25s IN A 192.149.252.21ns.apnic.net. 1d9h29m16s IN A 203.37.255.97
Tokyo
Amsterdam
Washington
Brisbane
7
373737© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Secondary DNS Example
• apnic.net zoneprimary DNS in Brisbane (ns.apnic.net)secondary DNS run by APNIC in Tokyo(svc00.apnic.net)zone secondaried by
RIPE NCC in AmsterdamARIN in Washington
Geographical and service provider redundancy – thisis the perfect example!
383838© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• Caching nameserverThis is the resolver – it is the DNS cacheYour customers use this as resolver, NOT your primaryor secondary DNSProvides very fast lookupsDoes NOT secondary any zonesOne, or preferably two per PoP (redundancy)Unix server, fast I/O, large amount of memory(512Mbytes+ depending on number of zones)
393939© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Caching Nameserver
To Core Routers
DIAL network
Web Cache
DNS Cache DNS Cache
Radius proxy
Switch redundancyRouter redundancyDNS Cache redundancy
DIAL users automatically given the IP addressesof DNS caches when they dial in
404040© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Anycasting the Caching Nameserver
• One trick of the tradeassign two unique IP addresses to befor the two DNS resolver systems
use these two IP addresses in every PoProute the two /32s across your backboneeven if the two resolver systems in the local PoP aredown, the IGP will ensure that the next nearestresolvers will be reachableKnown as IP Anycast
GeekAlert
414141© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• Efficient and resilient designPrimary DNS – keep it secureSecondary DNS – geographical and providerredundancy
Don’t ever put them on the same LAN, switched orotherwiseDon’t put them in the same PoP
Caching DNS – one or two per PoPreduces DNS traffic across backbonemore efficient, spreads the load
424242© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• SoftwareMake sure that the BIND distribution on the Unix systemis up to date
the vendor’s distribution is rarely current
Pay attention to bug reports, security issuesReboot the DNS cache on a regular (e.g. monthly) basis
clears out the cachereleases any lost RAMaccepted good practice by system administrators
8
434343© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:DNS
• ImplementationPut all your hosts, point-to-point links and loopbacksinto the DNS
under your ISP’s domain nameuse sensible/meaningful names
Put all your hosts, point-to-point links and loopbacksinto the REVERSE DNS also
don’t forget about in-addr.arpa – many ISPs dosome systems demand forward/reverse DNS mappingbefore allowing access
444444© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Mail
• Must have at least two mail hosts (MX records) forall supported domains
geographical separation helps
• POP3 server dedicated to that functionDIAL users get mail from here
• SMTP gateway dedicated to that functionDIAL users send mail via here
• Mail relay open to CUSTOMERS only!
454545© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Mail Example
• telstra.net mail (MX records)primary MX is mako1backup MX is postoffice – two addressesbackup MX used if primary unavailable
$ dig telstra.net mx
;; ANSWER SECTION:telstra.net. 1H IN MX 10 postoffice.telstra.net.telstra.net. 1H IN MX 5 mako1.telstra.net.
;; ADDITIONAL SECTION:postoffice.telstra.net. 1H IN A 139.130.4.7postoffice.telstra.net. 1H IN A 203.50.1.76mako1.telstra.net. 1H IN A 203.50.0.28
464646© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:Mail
• SoftwareMake sure that the MAIL and POP3 distributionson the Unix system are up to date
the vendor’s distribution are rarely current
Pay attention to bug reports, security issues,unsolicited junk mail complaints
IMPORTANT: Do NOT allow non-customers to use your mail system as a relay
474747© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:News
• News servers provide a Usenet news feed tocustomers
• Distributed design requiredIncoming newsfeed to one large serverDistributed to feed servers in each PoPFeed servers provide news feed to customersOutgoing news goes to another serverSeparate reading news systemSeparate posting news system
484848© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:News System Placement
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections News Collector
News Feeder
News Feeder
News Feeder
News Distributor
9
494949© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Services:News System Placement
POP One
POP Two
POP Three
Customerconnections
Customerconnections
Customerconnections
Externalconnections
Externalconnections News Collector
News Feeder
News Feeder
News Feeder
News Distributor505050© 2005, Cisco Systems, Inc. All rights reserved.
Cisco ISPWorkshops
ISP Services:News
• SoftwareMake sure that the Internet News distribution onthe Unix system is up to date
the vendor’s distribution is rarely current
Pay attention to bug reports, security issues,unsolicited junk posting complaints
IMPORTANT: Do NOT allow non-customers to use your news system for posting messages
51© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Addressing
525252© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Where to get IP addresses and AS numbers
• Your upstream ISP• Africa
AfriNIC – http://www.afrinic.net
• Asia and the PacificAPNIC – http://www.apnic.net
• North AmericaARIN – http://www.arin.net
• Latin America and the CaribbeanLACNIC – http://www.lacnic.net
• Europe and Middle EastRIPE NCC – http://www.ripe.net
535353© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ARIN
Internet Registry Regions
LACNIC
545454© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Getting IP address space
• Take part of upstream ISP’s PA spaceor
• Become a member of your Regional InternetRegistry and get your own allocation
Require a plan for a year aheadGeneral policies are outlined in RFC2050, more specificdetails are on the individual RIR website
• There is plenty of IPv4 address spaceregistries require high quality documentation
10
555555© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Addressing Plans – ISP Infrastructure
• Address block for router loop-backinterfaces
• Address block for infrastructureper PoP or whole backbonesummarise between sites if it makes senseallocate according to genuine requirements,not historic classful boundaries
565656© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Addressing Plans – Customer
• Customers assigned address spaceaccording to need
• Should not be reserved or assigned on aper PoP basis
ISP iBGP carries customer netsaggregation not required and usually notdesirable
575757© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Addressing Plans – ISP Infrastructure
Phase One220.10.0.0/21
Customer assignments Instrastructure Loopbacks
/24220.10.6.255220.10.0.1
220.10.0.0/20
Original assignments New Assignments
/24/24220.10.0.1
220.10.5.255 220.10.15.255
Phase Two
585858© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Addressing PlansPlanning
• Registries will usually allocate the nextblock to be contiguous with the firstallocation
Minimum allocation is /21
Very likely that subsequent allocation willmake this up to a /20So plan accordingly
595959© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Addressing Plans (contd)
• Document infrastructure allocationeases operation, debugging and management
• Document customer allocationcontained in iBGPeases operation, debugging and managementsubmit network object to RIR Database
60© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols
11
616161© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols
• IGP – Interior Gateway Protocolcarries infrastructure addresses, point-to-point linksexamples are OSPF, ISIS, EIGRP...
• EGP – Exterior Gateway Protocolcarries customer prefixes and Internet routescurrent EGP is BGP version 4
• No link between IGP and EGP
626262© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Why Do We Need an IGP?
• ISP backbone scalingHierarchyModular infrastructure constructionLimiting scope of failure
Healing of infrastructure faults using dynamicrouting with fast convergence
636363© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Why Do We Need an EGP?
• Scaling to large networkHierarchyLimit scope of failure
• PolicyControl reachability to prefixesMerge separate organizationsConnect multiple IGPs
646464© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Interior versus Exterior Routing Protocols
• Interiorautomatic neighbourdiscoverygenerally trust your IGProuters
prefixes go to all IGProutersbinds routers in one AStogether
• Exteriorspecifically configuredpeersconnecting withoutside networks
set administrativeboundariesbinds AS’s together
656565© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Interior versus Exterior Routing Protocols
• InteriorCarries ISPinfrastructureaddresses onlyISPs aim to keep theIGP small forefficiency andscalability
• ExteriorCarries customerprefixesCarries Internet prefixesEGPs are independentof ISP network topology
666666© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Hierarchy of Routing Protocols
BGP4and OSPF/ISIS
FDDI
Other ISPs
CustomersLocalIXP
BGP4 Static/BGP4
BGP4
12
676767© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols:Choosing an IGP
• Review the “Introduction to Link StateProtocols” presentation
i.e. – OSPF and ISIS have very similar properties
• ISP usually chooses between OSPF and ISISChoose which is appropriate for your operators’experienceIn IOS, both OSPF and ISIS have sufficient “nerdknobs” to tweak the IGP’s behaviour
686868© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols:IGP Recommendations
• Keep the IGP routing table as small as possibleIf you can count the routers and the point to point linksin the backbone, that total is the number of IGP entriesyou should see
• IGP details:Should only have router loopbacks, backbone WANpoint-to-point link addresses, and network addressesof any LANs having an IGP running on themStrongly recommended to use inter-routerauthenticationUse inter-area summarisation if possible
696969© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols:More IGP recommendations
• To fine tune IGP table size more, consider:Using “ip unnumbered” on customer point-to-pointlinks – saves carrying that /30 in IGP
(If customer point-to-point /30 is required formonitoring purposes, then put this in iBGP)Use contiguous addresses for backbone WAN links ineach area – can then summarise into backbone areaDon’t summarise router loopback addresses – as iBGPneeds thoseUse iBGP for carrying anything which does notcontribute to the Link State Routing process
707070© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols:iBGP Recommendations
• iBGP should carry everything whichdoesn’t contribute to the IGP routingprocess
Internet routing tableCustomer assigned addresses
Customer point-to-point linksDIAL network pools, passive LANs, etc
717171© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Routing Protocols:More iBGP Recommendations
• Scalable iBGP features:Use neighbour authentication
Use peer-groups to speed update process andfor configuration efficiencyUse communities for ease of filtering
Use route-reflector hierarchyRoute reflector pair per PoP (overlaid clusters)
Use route flap damping at the network edges
72© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Security
13
737373© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Security
• ISP Infrastructure security• ISP Network security• Security is not optional!• ISPs need to:
protect themselveshelp protect their customers from the Internetprotect the Internet from their customers
• The following slides are generalrecommendations
do more research on security before deploying anynetwork
747474© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Infrastructure Security
• router securityusernames, passwords, vty filters, TACACS+Disable telnet on vtys, only use SSHvty filters should only allow NOC access, noexternal accessSee IOS Essentials for the recommendedpractices for ISPs
757575© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Infrastructure Security
• ISP server securityusernames, passwords, TCP wrappers, IPTABLESprotect all servers using routers with strong filtersapplied
• Hosted services securityprotect network from hosted servers using routerswith strong filtersprotect hosted servers from Internet using routers withstrong filters
767676© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Infrastructure SecurityISP Server Protection
DNScache
DNSsecondary POP3 Mail
Relay NEWS
To core routers
Service NetworkGateway Routers
Access-list examples:
Allow tcp/established to all serversICMPDNS 2ary: udp/53 and tcp/53POP3: tcp/110Mail Relay: tcp/25 and ISP address
range onlyNews: tcp/119 and ISP
address range onlyDNS Cache: udp/53Web server: tcp/80
Other necessary filters:
All servers: SSH (tcp/22) from NOC LAN only
Webserver
777777© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Access-list examples:InboundAllow tcp/established to all serversICMPWeb server: tcp/80SSH for customer accessAny other ports for services
sold to customers
OutboundICMPAllow DNS udp/53 and
tcp/53Block all access to ISP
address range
ISP Infrastructure SecurityHosted Server Protection
Server5Server1 Server2 Server3 Server4
To core routers
Service NetworkGateway Routers
Server6
787878© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Infrastructure Security
• premises securitylocks – electronic/card key preferredsecure access – 24x7 security arrangementsenvironment control – good aircon
• staff responsibilitypassword policy, strangers, temp staffemployee exit procedures
• RFC2196(Site Security Handbook)
• RFC3871(Operational Security Requirements for Large ISP IPNetwork Infrastructure )
14
797979© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Network Security
• Denial of Service Attackseg: “smurfing”see http://www.denialinfo.com
• Effective filteringnetwork borders – see Cisco ISP Essentialscustomer connections – unicast RPFnetwork operation centreISP corporate network – behind firewall
808080© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Network SecuritySecure external access
• How to provide staff access from outsideset up ssh gateway (Unix system with ssh daemon andnothing else configured)provide ssh client on all staff laptopsssh available on Unix and Windowsssh is Secure Shell – encrypted link
• How not to provide access from outsidetelnet, rsh, rlogin – these are all insecureopen host – insecure, can be compromised
818181© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Ingress & Egress Route Filtering
Your customers should not besending any IP packets out to the
Internet with a source addressother then the address you have
allocated to them!
82© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Out of Band Management
838383© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Out of Band Management
• Not optional!• Allows access to network equipment in times of
failure
• Ensures quality of service to customersminimises downtimeminimises repair timeeases diagnostics and debugging
848484© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Out of Band Management
• OoB Example – Access server:modem attached to allow NOC dial inconsole ports of all network equipment connected toserial portsLAN and/or WAN link connects to network core, or viaseparate management link to NOC
• Full remote control access under allcircumstances
15
858585© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Out of Band Network
Ethernetto the NOC
Router, switchand ISP server
consoles
(Optional) Out of bandWAN link to other PoPs
Modem – accessto PSTN for out of
band dialin
Equipment RackEquipment Rack
868686© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Out of Band Management
• OoB Example – Statistics gathering:Routers are NetFlow and syslog enabled
Management data is congestion/failure sensitiveEnsures management data integrity in case of failure
• Full remote information under all circumstances
87© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Test Laboratory
888888© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Test Laboratory
• Designed to look like a typical PoPoperated like a typical PoP
• Used to trial new services or newsoftware under realistic conditions
• Allows discovery and fixing of potentialproblems before they are introduced tothe network
898989© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Test Laboratory
• Some ISPs dedicate equipment to the lab
• Other ISPs “purchase ahead” so thattoday’s lab equipment becomestomorrow’s PoP equipment
• Other ISPs use lab equipment for “hotspares” in the event of hardware failure
909090© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Test Laboratory
• Can’t afford a test lab?Set aside one spare router and server to trial new services
Never ever try out new hardware, software or services onthe live network
• Every major ISP in the US and Europe has a test labIt’s a serious consideration
16
91© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Operational Considerations
929292© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Operational Considerations
Why design the world’s best networkwhen you have not thought about whatoperational good practices should be
implemented?
939393© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Operational ConsiderationsMaintenance
• Never work on the live network, no matter howtrivial the modification may seem
Establish maintenance periods which your customers areaware of
e.g. Tuesday 4-7am, Thursday 4-7am
• Never do maintenance on a FridayUnless you want to work all weekend cleaning up
• Never do maintenance on a MondayUnless you want to work all weekend preparing
949494© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Operational ConsiderationsSupport
• Differentiate between customer support and theNetwork Operations Centre
Customer support fixes customer problemsNOC deals with and fixes backbone and Internet relatedproblems
• Network Engineering team is last resortthey design the next generation network, improve therouting design, implement new services, etcthey do not and should not be doing support!
959595© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
Operational ConsiderationsNOC Communications
• NOC should know contact details forequivalent NOCs in upstream providersand peers
• Or consider joining the INOC-DBA systemVoice over IP phone system using SIPRuns over the Internetwww.pch.net/inoc-dba for more information
96© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Network Design
Summary
17
979797© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Design Summary
• KEEP IT SIMPLE & STUPID ! (KISS)
• Simple is elegant is scalable
• Use Redundancy, Security, andTechnology to make life easier for yourself
• Above all, ensure quality of service foryour customers
98© 2005, Cisco Systems, Inc. All rights reserved.Cisco ISPWorkshops
ISP Network DesignISP/IXP WorkshopsISP/IXP Workshops