dren ipv6 implementation update

21-Jan-2008 DREN IPv6 Update 1

DREN IPv6 Implementation Update

Techs in Paradise, 2008Jan 21, 2008Honolulu, HI

Ron BroersmaDREN Chief Engineer

High Performance Computing Modernization [email protected]


Background

• WAN provider for the DoD R&D community

• Also serving as DoD’s IPv6 pilot implementation of the DoD CIO June ’08 mandate

• Deploying IPv6 where possible in a production environment– See what works and what’s broken– See what’s missing– Share lessons learned


Previously• Reported to you previously:

– Most serious problem is lack of IPv6 support in many security products (firewall, IDS, IPS, VPNs, web proxy, etc).

• Major inhibitor to deployment of IPv6 in protected enclaves– Numerous bugs

• hurts deployment and require workarounds– Increased complexities with running dual-stack– Vista missing some v6 support– Some things getting better

• Important fixes in key software components• Some incentives from Asian demands

– The scare over RH0, now deprecated– Router meltdown– Proponents and providers aren’t eating their own

dogfood


What’s new• Still finding bugs, but they are getting harder

to diagnose• Increased campus deployment efforts• Expanded peering• Testbed downsizing• Tunnel brokers updated• www.v6.dren.net web site restored and

updated– Moved to new server (virtual)– Fedora 7– Some cleanup and restructuring of content

• DHPCv6 interoperability testing• Vendors still aren’t eating their own dogfood

http://www.v6.dren.net/


DREN IPv6 Testbed

• Starting to shut down DRENv6 testbed– 3 (of 9) Core nodes shut down

• ERDC• NAVO• AFRL Kirtland

– Moving Peering from testbed to production network

• SPRINT• UUNET• NTT/Verio• … and others in the works


DRENv6 “testbed”Logical Topology

Dayton

San Diego

Albuquerque

Wash D.C.

Stennis

Vicksburg

Aberdeen

ATM PVC (OC-3)

tunnel

HICv6

(Hawaii)

GlobalCrossing

HurricaneElectric

LAVAnet

SPRINT

vBNS+

6TAP

SSC CharlestonSSAPAC

SSC San Diego

WCISD

AOL

NRL

ARLWPAFB

ERDC

NAVO

C&W

Cisco

NTTComVerio

AFRLKirtland AFB

Abilene

SD-NAPSDSC

Core Router

“site”

IXP

ISP orBGP Neighbor

FIX-West Abilene

HP

AIX-v6

TIC

JITC

Tunnel broker


Peering• Upgraded all NIDS at peering locations to insure visibility

of IPv6 traffic (security).• Renewed effort to improve peering

– Production network only had 8 operational as of 2 months ago.

– Move peers from testbed to production• Production network used testbed for transit to most peers.

– Add new peers• UUNET (WAE and HAY)• TWAREN (Seattle, working on Starlight)• SPRINT (MAEwest)• NTT/Verio (vastly improved performance to Japan sites0• NLR (Seattle, MAEwest)• Hurricane Electric (WAE tunnel 1/22/08, MAE west cross connect being worked)• UH (last week)• HIX and LavaNet in the works• USGS (this week)• Qwest (working final draft of agreement)

• Had to adjust policy– Waive requirement to peer at 3 locations– Waive requirement for high bandwidth native peering


Path from UH to DREN

dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.nettraceroute6 to narf.v6.dren.net (2001:480:10:1050::5) from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets 1 2607:f278:5:3::1 2.873 ms 3.163 ms 5.251 ms 2 2607:f278:0:5::2 19.21 ms 6.222 ms 12.63 ms 3 so-2-1-0.bb1.a.syd.aarnet.net.au 95.954 ms 103.687 ms 102.948 ms 4 ge-0-0-0.bb1.b.syd.aarnet.net.au 107.493 ms 95.859 ms 100.499 ms 5 so-2-1-0.bb1.b.sea.aarnet.net.au 264.709 ms 259.346 ms 259.667 ms 6 dren-1-lo-jmb-706.sttlwa.pacificwave.net 157.101 ms 160.606 ms 170.337 ms 7 so12-0-0-0.sandiego.dren.net 189.137 ms 189.147 ms 186.446 ms 8 narf.v6.dren.net 186.873 ms 190.175 ms 207.715 ms

dchp-166-122-119-217:~ ron$ traceroute6 www.v6.dren.nettraceroute6 to narf.v6.dren.net (2001:480:10:1050::5) from 2607:f278:5:3:217:f2ff:fee8:66db, 30 hops max, 12 byte packets 1 2607:f278:5:3::1 2.927 ms 2.413 ms 4.273 ms 2 2607:f278:0:1::1 9.538 ms 18.885 ms 2.41 ms 3 2001:480:0:c81::1 17.07 ms 2.736 ms 3.02 ms 4 2001:480:0:c2f::1 54.571 ms 57.151 ms 54.373 ms 5 so12-0-0-0.sandiego.dren.net 88.173 ms 90.586 ms 84.086 ms 6 narf.v6.dren.net 84.489 ms 84.001 ms 84.304 ms

Before…

After…


Recent Frustrations

• Products still not doing IPv6– Juniper NSM - slipped 18 months– Tipping Point - slipped 18 months– Bluecoat web/proxy - still no beta code, although

promised a year ago

• Bugs, bugs, bugs– Netscreen ND is broken– BIND sometimes stops responding when using IPv6

transport– Losing first packet breaks NTP– ping6 annoyance– … and more


Netscreen ND bug

No. Time Source Destination Protocol Info 1 0.000000 fe80::211:92ff:fe10:ca90 ff02::1:ff00:1 ICMPv6 Neighbor solicitation 2 0.001721 fe80::212:1eff:feaf:9906 fe80::211:92ff:fe10:ca90 ICMPv6 Neighbor advertisement

Internet Control Message Protocol v6 Type: 136 (Neighbor advertisement) Code: 0 Checksum: 0x5e5b [correct] Flags: 0x00000060 0... .... .... .... .... .... .... .... = Not Router .0.. .... .... .... .... .... .... .... = Not Solicited ..0. .... .... .... .... .... .... .... = Not Override Target: 2001:480:822:1f01::1 ICMPv6 Option (Target link-layer address) Type: Target link-layer address (2) Length: 8 Link-layer address: 00:12:1e:af:99:06


Neighbor Advertisement

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Code | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |R|S|O| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + Target Address + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options ... +-+-+-+-+-+-+-+-+-+-+-+-

S Solicited flag. When set, the S-bit indicates that the advertisement was sent in response to a Neighbor Solicitation from the Destination address. The S-bit is used as a reachability confirmation for Neighbor Unreachability Detection. It MUST NOT be set in multicast advertisements or in unsolicited unicast advertisements.


Switch loses first packet

• Foundry BigIron will lose first packet after period of no activity

When the mac address of ins1.sd is not in the mac table of the BigIron....

[[email protected] ~]$ ping6 ins1.sdPING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=0.778 ms64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=2 ttl=64 time=1.87 ms

When the mac address is in the table....

[[email protected] ~]$ ping6 ins1.sdPING ins1.sd(ins1.sd.spawar.navy.mil) 56 data bytes64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=0 ttl=64 time=0.789 ms64 bytes from ins1.sd.spawar.navy.mil: icmp_seq=1 ttl=64 time=4.85 ms

Notice the first time that it loses sequence 0.

mailto:[email protected]

mailto:[email protected]


Losing first packet

First there is the IPv6 neighbor discovery, which makes it through and is answered because it is sent to a ethernet multicast address..

17:57:26.913312 00:0b:db:93:b5:73 > 33:33:ff:00:00:02, 2001:480:10:4::3 > ff02::1:ff00:2: icmp6: neighbor sol: who has 2001:480:10:4::217:57:26.915433 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: neighbor adv: tgt is 2001:480:10:4::2

By now the mac address (00:0b:db:93:b5:85) should be in the mac table (but I think it isn't yet, due to some internal delay). The next packet is the one that never makes it through the BigIron...

17:57:26.915457 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 0

That echo request is never replied to. But, the next one makes it OK...

17:57:27.912144 00:0b:db:93:b5:73 > 00:0b:db:93:b5:85, 2001:480:10:4::3 > 2001:480:10:4::2: icmp6: echo request seq 117:57:27.912901 00:0b:db:93:b5:85 > 00:0b:db:93:b5:73, 2001:480:10:4::2 > 2001:480:10:4::3: icmp6: echo reply seq 1

Is it possible that 24 microseconds (between the NA and the echo request packet) isn't enough time for the BigIron to get the mac table updated?


Losing first packet: impact

• NTP fails– Normally backs off to 1024 second updates– Single UDP packet sent on update– Client never sees packet, declares server

down, and restart at 64 seconds, backs off, then fails again


ping6 annoyance

• ping6 does a DNS lookup on EVERY packet– Bug in addrconf patch– The value was never

copied to the cache

--- iputils/ping.c.addrcache 2002-09-20 17:08:11.000000000 +0200 +++ iputils/ping.c 2003-05-15 16:41:19.000000000 +0200 @@ -1124,6 +1124,12 @@ { struct hostent *hp; static char buf[4096]; + static __u32 addr_cache = 0; + + if ( addr == addr_cache ) + return buf; + + addr_cache = addr;

if ((options & F_NUMERIC) || !(hp = gethostbyaddr((char *)&addr, 4, AF_INET))) --- iputils/ping6.c.addrcache 2002-09-20 17:08:11.000000000 +0200 +++ iputils/ping6.c 2003-05-15 16:41:19.000000000 +0200 @@ -893,7 +893,14 @@ */ char * pr_addr(struct in6_addr *addr) { - struct hostent *hp = NULL; + static struct hostent *hp = NULL; + static struct in6_addr addr_cache = {{{0,0,0,0}}}; + + if ( addr->s6_addr32[0] == addr_cache.s6_addr32[0] && + addr->s6_addr32[1] == addr_cache.s6_addr32[1] && + addr->s6_addr32[2] == addr_cache.s6_addr32[2] && + addr->s6_addr32[3] == addr_cache.s6_addr32[3] ) + return hp ? hp->h_name : pr_addr_n(addr);

if (!(options&F_NUMERIC)) hp = gethostbyaddr((__u8*)addr, sizeof(struct in6_addr), AF_INET6);


Expanded deployment• Many campus LANs now 100% IPv6 enabled• One site has achieved dual-stacking ALL desktops

and servers (except 2)– Others have active programs to do the same, along with

getting help desk trained and tooled for supporting it.

• Remove “A” record from DNS for servers– Possible when all clients for that server are dual-stack

• Servers running v6-only (no IPv4 address on any interface, nor on network it is connected to)– Fedora 7 - some manual fiddling of the config is

required, but works– DNS issues (BIND sometimes stops responding when

doing v6 transport queries)


Fedora 7 IPv6-only

• Connect to IPv6-only LAN, so no cheating.

• Configure from GUI, like any point-and-click sysadmin would do.

• Getting it to actually work…– Manual hacking of ifcfg-eth0

• Delete IPV6ADDR and IPV6PREFIX• Set ONBOOT = yes

– Fix broken /etc/hosts– Configure sendmail to listen on ::1– Fix /etc/yum.repos.d/* files to point to an

IPv6-capable mirror


IPv6-enablement milestones

and scoring (proposed)1. IPv6 address allocation and address/routing plan developed2. LAN (wired and wireless) is fully v6-enabled (all routers do v6 on all interfaces) and is

connected to the IPv6 Internet (WAN).– The implication is that any v6-enabled device can connect anywhere in the LAN and get IPv6 Internet

connectivity. – (worry about routing implementation, make sure address plan is right, think about security and

performance, play with multicast, make sure network staff is trained to support it, etc)3. Internal infrastructure services (DNS, NTP, DHCP, SMTP, etc) are IPv6-enabled4. Public facing services (public DNS, MXs, public web site) are IPv6-enabled5. Network management infrastructure (management LAN, SNMP, SYSLOG, MIBs, management

access to switches/routers/etc) is IPv6-enabled6. Most workstations and servers are v6-enabled

– (behind this is the support infrastructure, i.e. help desk and other IT support, and a message to sys admins to turn on IPv6 where possible, and servers have AAAA records in DNS, and your help desk tools/scripts for things like host registration and update are upgraded to handle IPv6 addresses

7. Once critical mass is reached on the client side, remove "A" records for servers (resulting in final incentive and some pain for those that didn't dual-stack their workstations).

– You don't need to do them all at once, just one at a time as their clients all become dual-stack8. Migrate some network segments to IPv6-only, with no IPv4 addresses anywhere

– (this forces one to figure out how to make hosts operate in a v6-only environment, helps the organization figure out what's missing, forces the implementation of some kind of transition mechanism to bridge to the IPv4-only world, plus adds continued incentive to get more stragglers upgraded since they can't even get there by typing the IP address

9. Deploy advanced features (mobile-ip, v6 multicast, etc.), provide remote IPv6 access (travellers, telecommuters, home, etc) from v4-only environments, cleanup and reduce complexity (readdressing network if necessary), ....

10. Declare victory• you claim a perfect score in public, and are willing to stand up to scrutiny

Scoring: Scale of 0 to 10. 1 point for any milestone that is 95% complete.


Commitment to IPv6?

• Are network product vendors really committed to IPv6 support?

• Are they using it in their production networks?

• Do they have an IPv6 presence on the Internet?

• Do they follow the “eat your own dogfood” principle?

• A survey…


Vendor scorecard

• Looked in DNS to see if there were AAAA records for www, MX, and DNS.

• Quick sampling of major computer and network companies showed no public facing IPv6.

• 6 months ago, and then today.

Jan 2008

June 2007


Scorecard – IPv6 Summit Sponsors (March ’07)

• Grand and Gold Sponsors of 2007 IPv6 Summit.

• Only one has an IPv6 presence at their corporate “front door”


Summary: Situation Today

• DREN has been successfully using IPv6 in a production environment, with many dual-stack systems and services, for years– Modern operating systems just work, out of the box

(MacOSX, Linux, Vista, Solaris 10, etc)

• Most urgent needs from our perspective:– Need parity with IPv4 in all implementations– Enabling IPv6 must NOT break things– Need to make security stacks fully IPv6 capable

• Firewalls, IDS, proxies, IDP/IPS, ACLs– Eat your own dogfood

• The rest of DoD will not make the June ’08 deadline.

• The US is falling far behind Asia and other regions.– Prevalent attitude: IPv6 just another GOSIP, or ADA, or ….– A call to action


END

dren ipv6 implementation update

Documents

deployment of ipv6

ipv6 support

mandatedeploying ipv6

dods ipv6 pilot implementation

new peersuunet wae

web site

improved performance

locationswaive requirement