dynamic multipoint vpn (dmvpn) troubleshooting · pdf filescenario #1 solutions when we run...

Dynamic Multipoint VPN (DMVPN)

Troubleshooting

Solutions

Luke Bibby, CCIE #45527

Quick Overview of Scenario Solutions

Scenario #1

Incorrect NHRP mapping on R2

No NHS configuration on R3

Scenario #2

Wrong NRHP authentication key on R2

NHRP Shortcut and redirect on the wrong routers

Scenario #3

Incorrect transform set on R3

Incorrect ISAKMP key on R2

Scenario #4

Incorrect NHRP mappings on R2 and R3

No NHS configuration on R2

Scenario #5

R1, R2 and R3 have the wrong ESP mode

R4 has no NAT configuration for UDP/4500

R2 has the wrong ISAKMP key peer

R3 is missing the IPsec protection profile under the DMVPN interface

Scenario #1 Solutions

When we run the following command we notice that R1 does not know about any of the DMVPN

spokes (this is the same output as from the scenario):

R1#show dmvpn

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete

N - NATed, L - Local, X - No Socket

# Ent --> Number of NHRP entries with same NBMA peer

NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting

UpDn Time --> Up or Down Time for a Tunnel

Let’s check the NRHP cache on the spokes R2 and R3 to see if they have a mapping to the hub.

R2#show ip nhrp

10.1.1.1/32 via 10.1.1.1

Tunnel100 created 00:04:47, never expire

Type: static, Flags:

NBMA address: 192.168.123.1

Well a mapping exists to R1 so let’s assume that this is not the problem. Can R2 ping R1’s NBMA

address when sourced from its own NBMA address? We will also run a generalised NRHP debug to

see if there are hints on the problem.

R2#debug nhrp

NHRP protocol debugging is on

R2#ping 192.168.123.1 source e0/0 repeat 1

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 192.168.123.1, timeout is 2 seconds:

Packet sent with a source address of 10.0.12.2

*Dec 7 00:36:08.506: NHRP: NHRP could not map 192.168.123.1 to NBMA, cache entry

not found

*Dec 7 00:36:08.506: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel100 netid-

out 132

*Dec 7 00:36:08.506: NHRP: Checking for delayed event NULL/192.168.123.1 on list

(Tunnel100).

*Dec 7 00:36:08.506: NHRP: No node found..

Success rate is 0 percent (0/1)

R2#

*Dec 7 00:36:16.207: NHRP: Checking for delayed event NULL/192.168.123.1 on list

(Tunnel100).

*Dec 7 00:36:16.207: NHRP: No node found.

*Dec 7 00:36:16.207: NHRP-ATTR: Requester Ext Len: Total ext_len with NHRP

attribute VPE 20

*Dec 7 00:36:16.207: NHRP: Sending NHRP Resolution Request for dest: 192.168.123.1

to nexthop: 192.168.123.1 using our src: 192.168.123.2

*Dec 7 00:36:16.207: NHRP: Attempting to send packet via DEST 192.168.123.1

*Dec 7 00:36:16.207: NHRP: NHRP could not map 192.168.123.1 to NBMA, cache entry

not found

*Dec 7 00:36:16.207: NHRP: Send Resolution Request via Tunnel100 vrf 0, packet

size: 72

*Dec 7 00:36:16.207: src: 192.168.123.2

R2#, dst: 192.168.123.1

*Dec 7 00:36:16.207: NHRP: Encapsulation failed for destination 192.168.123.1 out

Tunnel100

The above highlighted line is indicating a problem with the mapping to R1. Let’s check the IP

addresses configured on R2 to see if there is an addressing problem.

R2#show ip int br | exc una

Interface IP-Address OK? Method Status

Protocol

Ethernet0/0 10.0.12.2 YES manual up up

Tunnel100 192.168.123.2 YES manual up up

We can now see the problem when we look at the IP address configured on R2’s tunnel address. We

saw earlier that R2 had a mapping to R1’s NBMA address of 192.168.123.1, which is in the same

network as R2’s tunnel address. This is leading us towards thinking that there is a problem with R2’s

NHRP mapping. Let’s see what the configuration on R2’s tunnel looks like.

R2#show run int tun100

Building configuration...

Current configuration : 258 bytes

!

interface Tunnel100

ip address 192.168.123.2 255.255.255.0

no ip redirects

ip nhrp map 10.1.1.1 192.168.123.1

ip nhrp map multicast 10.1.1.1

ip nhrp network-id 132

ip nhrp nhs 192.168.123.1

tunnel source Ethernet0/0

tunnel mode gre multipoint

end

And now we’ve confirmed the NRHP map is the problem. Let’s fix this on R2.

R2(config)#int tun100

R2(config-if)#no ip nhrp map 10.1.1.1 192.168.123.1

R2(config-if)#ip nhrp map 192.168.123.1 10.1.1.1

Let’s now try to ping R1’s tunnel interface.

R2#ping 192.168.123.1



..!!!

Success rate is 60 percent (3/5), round-trip min/avg/max = 1/1/1 ms

We should see on R1 that R2 now appears as a DMVPN spoke.

R1#show dmvpn

Legend: Attrb S – Static, D – Dynamic, I – Incomplete

N – NATed, L – Local, X – No Socket

# Ent Number of NHRP entries with same NBMA peer

NHS Status: E Expecting Replies, R Responding, W Waiting

UpDn Time Up or Down Time for a Tunnel

Interface: Tunnel100, Ipv4 NHRP Details

Type:Hub, NHRP Peers:1,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb

----- --------------- --------------- ----- -------- -----

1 10.0.12.2 192.168.123.2 UP 00:00:50 D

R3 is still not showing up as DMVPN spoke however so let’s see if the mapping is wrong there too.

R3#show ip nhrp

192.168.123.1/32 via 192.168.123.1




This looks fine. Can R3 ping R1’s NBMA address when sourcing the ping from it’s own NBMA

address?

R3#ping 10.1.1.1 source e0/0




!!!!!


This proves that there is no underlying routing problem between the tunnel source endpoints. Can

R3 ping R1’s tunnel address?

R3#ping 192.168.123.1



.....


As mostly expected this does not work (otherwise R1 would have seen R3 as a DMVPN spoke

earlier). Let’s debug NHRP to see if we can isolate the problem.

R3#ping 192.168.123.1 repeat 1



*Dec 7 00:51:49.697: NHRP: Setting ‘used’ flag on cache entry with nhop:

192.168.123.1

*Dec 7 00:51:49.698: NHRP: NHRP successfully mapped ‘192.168.123.1’ to NBMA

‘10.1.1.1’.


The output seems a lot more quiet than what we saw when we did the same on R2. Notably, there

are no NHRP Registration Request messages bent sent out, which would normally be sent to the

NHRP NHS so that R3 can register itself to the hub router. Let’s see what NHRP NHS is configured on

R3’s tunnel interface.




!

interface Tunnel100

ip address 192.168.123.3 255.255.255.0

no ip redirects

ip nhrp map 192.168.123.1 10.1.1.1





end

And now we can see the problem: R3 is a spoke but does not have an NHS configured so where is it

supposed to send the NHRP Resolution Request messages to? Let’s fix this.

R3#conf t


R3(config-if)#ip nhrp nhs 192.168.123.1

Let’s see now on R1 if R3 has successfully registered itself to the DMVPN.

R1#show dmvpn

Legend: Attrb S – Static, D – Dynamic, I – Incomplete

N – NATed, L – Local, X – No Socket

# Ent Number of NHRP entries with same NBMA peer

NHS Status: E Expecting Replies, R Responding, W Waiting

UpDn Time Up or Down Time for a Tunnel

Interface: Tunnel100, Ipv4 NHRP Details



----- --------------- --------------- ----- -------- -----

1 10.0.12.2 192.168.123.2 UP 00:13:41 D

1 10.0.13.3 192.168.123.3 UP 00:02:40 D

This looks much better! The scenario told us to verify that R2 could ping R3 so let’s now prove that

our fixes have resolved the original problems.

R2#ping 192.168.123.3



!!!!!


And we should also see an entry for R3 in the NHRP cache on R2.

R2#show ip nhrp

192.168.123.1/32 via 192.168.123.1


Type: static, Flags: used


192.168.123.2/32 via 192.168.123.2

Tunnel100 created 00:00:04, expire 01:59:55

Type: dynamic, Flags: router unique local


(no-socket)

192.168.123.3/32 via 192.168.123.3


Type: dynamic, Flags: router



The scenario indicated that R2 is having problems registering itself to R1 so let’s first check the NRHP

mapping on R2.

R2#show ip nhrp

192.168.123.1/32 via 192.168.123.1




According to the diagram it looks fine. Let’s see if R2 can ping R1’s NBMA address sourced from it’s

own NBMA address.

R2#ping 192.168.123.1 source e0/0




!!!!!


No problems there. Let’s run a generalised NHRP debug and try and ping over the tunnel.

R2#debug nhrp


R2#

R2#ping 192.168.123.1 rep 1



*Dec 7 01:16:34.367: NHRP: Setting 'used' flag on cache entry with nhop:

192.168.123.1

*Dec 7 01:16:34.367: NHRP: NHRP successfully mapped '192.168.123.1' to NBMA

'10.1.1.1'

*Dec 7 01:16:35.958: NHRP: Setting retrans delay to 64 for nhs dst 192.168.123.1

*Dec 7 01:16:35.958: NHRP-ATTR: Requester Ext Len: Total ext_len with NHRP

attribute VPE 55

*Dec 7 01:16:35.958: NHRP: Attempting to send packet via DEST 192.168.123.1


192.168.123.1


'10.1.1.1'

*Dec 7 01:16:35.958: NHRP: Encapsulation succeeded. Sending NHRP Control Packet

NBMA Address: 10.1.1.1

*Dec 7 01:16:35.958: NHRP: Send Registration Request via Tunnel100 vrf 0, packet

size: 107

*Dec 7 01:16:35.958: src: 192.168.123.2.


R2#, dst: 192.168.123.1

*Dec 7 01:16:35.958: NHRP: 131 bytes out Tunnel100

This confirms that there is no mapping problems, and an NHS must be configured otherwise no

NHRP Registration Request would be sent. Let’s see if R1 has an NHRP cache entry for R2.

R1#show ip nhrp

192.168.123.2/32


Type: incomplete, Flags: negative

Cache hits: 7

It has an entry but it is incomplete. So if R2 can successfully encapsulate packets for the tunnel, and

R1 is receiving them (so it shouldn’t be a filtering issue) but is marking them as complete, what else

could be wrong? Let’s check the configuration of the tunnel on R2’s end.




!

interface Tunnel100

ip address 192.168.123.2 255.255.255.0

no ip redirects

ip nhrp authentication luke123


ip nhrp map 192.168.123.1 10.1.1.1


ip nhrp nhs 192.168.123.1



end

We can see that authentication is in use for NHRP. If there was an NHRP authentication issue this

could indicate why R1 is marking the NRHP cache as incomplete. Let’s see what key R1 has

configured on the tunnel.

R1#show run int tun100 | inc auth

ip nhrp authentication 1uke123

If you look carefully, R2’s key is “luke123” and R1’s key is “1uke123”. Seeing as we don’t know which

is the correct key, let’s check what R3’s configured key is.

R3#show run int tun100 | inc auth


So R1 and R3 have the correct key, so let’s change the key on R2.


R2(config-if)#ip nhrp authentication 1uke123

Soon after configuring this you should see the following message printed to the console indicating

that EIGRP has now formed an adjacency over the tunnel.

R2#

*Dec 7 01:30:30.988: %DUAL-5-NBRCHANGE: EIGRP-IPv4 123: Neighbor 192.168.123.1

(Tunnel100) is up: new adjacency

Now let’s see if we are receiving EIGRP routes from R1 now.

R2#show ip route eigrp

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.33.0/32 is subnetted, 1 subnets

D 192.168.33.3 [90/102400640] via 192.168.123.1, 00:00:14, Tunnel100

And let’s see if R3 is receiving an EIGRP route to R2’s loopback over the tunnel.












D 192.168.22.2 [90/102400640] via 192.168.123.1, 00:01:31, Tunnel100

So the control plane appears to be setup correctly for the two spokes to be able to ping each other’s

loopbacks, so let’s try it out.

R2#ping 192.168.33.3 source loop192




!!!!!


Great! Now let’s see if we can match the verification output from the troubleshooting scenario with

a traceroute.

R2#traceroute 192.168.33.3 source loop192 num


Tracing the route to 192.168.33.3

VRF info: (vrf in name/id, vrf out name/id)

1 192.168.123.1 1 msec 0 msec 1 msec

2 192.168.123.3 1 msec * 1 msec

We are still seeing R1 as a hop, so let’s try the same traceroute again (the original scenario said we

may have to try this a few times).

R2#traceroute 192.168.33.3 source loop192 num




1 192.168.123.1 1 msec 0 msec 0 msec

2 192.168.123.3 1 msec * 1 msec

There are a few ways to fix this:

1. Set “no next-hop-self” under the EIGRP process on R1

2. Use NHRP next-hop override (NHO)

If you look back to the original scenario, the EIGRP route on R2 to R3’s looback had the percent

symbol next to the route, indicating that NHRP has overridden the next-hop. We can confirm that

this is true if we look at the route codes at the top of the routing table output.

R2#show ip route | inc \%


Let’s look at the current next-hop for R3’s loopback from R2’s perspective.












D 192.168.33.3 [90/102400640] via 192.168.123.1, 03:04:29, Tunnel100

For NHO to work a few things need to be in place:

1. The hub router, R1 in this case, needs to be configured to send the NHRP Redirect Indicator

when it notices that ingress and egress interface are the same for traffic

2. The spoke routers, R2 and R3 in this case, needs to be configured to install the shortcut, or

rather, the NHO

Let’s use the generalised NHRP debug to see if R1 is actually sending the NHRP redirect indicator

when we ping between the loopbacks on R2 and R3.

R2#debug nhrp


R2#ping 192.168.33.3 source loopback192 rep 1




!


R2#


192.168.123.1


'10.1.1.1'

This is not what we want: we need to see R1 send the NHRP redirect so that we can install the NHO.

Let’s see if R1 is configured to send this.

R1#sh run int tun100



!

interface Tunnel100

description DMVPN

ip address 192.168.123.1 255.255.255.0

no ip redirects


ip nhrp map multicast dynamic


ip nhrp shortcut

tunnel source Loopback0


end

In this case, R1 has been configured to install an NHO rather to send the NHRP redirects. It won’t

actually affect the operation if we leave this here and add the configuration to send NHRP redirects,

but on the CCIE lab it is plausible that you could be penalized for having additional unnecessary

configuration so in my opinion is better to take this line out and add in the right configuration.


R1(config-if)#no ip nhrp shortcut

R1(config-if)#ip nhrp redirect

Let’s try the traceroute again on R2 again and also leave the generalised NHRP debug on.

R2#trace 192.168.33.3 source loop192 num




1 192.168.123.1 0 msec 1 msec 0 msec

2 192.168.123.3 1 msec


192.168.123.1


'10.1.1.1'


192.168.123.1


'10.1.1.1'


192.168.123.1


'10.1.1.1'


192.168.123.1


'10.1.1.1'

*Dec 7 04:50:33.269: NHRP-ATTR: ext_type: 32772, ext_len : 0





*Dec 7 04:50:33.269: NHRP: Receive Traffic Indication via Tunnel100 vrf 0, packet

size: 99

*Dec 7 04:50:33.269: NHRP: netid_in = 132, to_us = 0

*Dec 7 04:50:33.269: NHRP: nhrp_rtlookup yielded Loopback192

*Dec 7 04:50:33.269: NHRP: netid_out 0, netid_in 132

*Dec 7 04:50:33.269: NHRP: Parsing NHRP Traffic Indication


192.168.123.1


'10.1.1.1'

This is also not what we want! We should that after receiving the NRHP redirect that R2 sends an

NHRP Resolution Request for 192.168.33.3 to try and identify the correct spoke to send this traffic to

directly. Let’s see if shortcut is configured on R2 and R3.

R2#show run int tun100 | inc shortcut


Well this is our next probem; shortcut is not configured. Let’s configure this on R2 and R3.


R2(config-if)#ip nhrp shortcut


R3(config-if)#ip nhrp shortcut

Now let’s try our traceroute again.





1 192.168.123.3 1 msec * 5 msec

This is what we want! We are sending traffic directly to R3 now. Let’s also confirm R2’s routing table

for R3’s loopback to see if we have the NHO code next to the route.












D % 192.168.33.3 [90/102400640] via 192.168.123.1, 00:09:11, Tunnel100


In the original scenario we were shown something similar to the following output on R1.

R1#show crypto session

Crypto session current status

Interface: Tunnel100

Session status: UP-ACTIVE

Peer: 10.0.12.2 port 500

IKEv1 SA: local 10.1.1.1/500 remote 10.0.12.2/500 Active

IPSEC FLOW: permit 47 host 10.1.1.1 host 10.0.12.2

Active SAs: 4, origin: crypto map

nterface: Ethernet0/1

Session status: UP-IDLE

Peer: 10.0.13.3 port 500


The first thing we can notice about the output is that R3 has established a pair of IPsec SAs with R2

but not with R3. Let’s run an IPsec debug on R1 to see if that highlights any errors for us.

R1#debug crypto ipsec

Crypto IPSEC debugging is on

Then restart the crypto sessions on R3 using the following command.

R3#clear crypto session

Then observe the debug output on R1.

R1#

*Dec 7 05:10:24.737: IPSEC(validate_proposal_request): proposal part #1

*Dec 7 05:10:24.737: IPSEC(validate_proposal_request): proposal part #1,

(key eng. msg.) INBOUND local= 10.1.1.1:0, remote= 10.0.13.3:0,

local_proxy= 10.1.1.1/255.255.255.255/47/0,

remote_proxy= 10.0.13.3/255.255.255.255/47/0,

protocol= ESP, transform= NONE (Transport),

lifedur= 0s and 0kb,

spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0

*Dec 7 05:10:24.738: map_db_find_best did not find matching map

*Dec 7 05:10:24.738: IPSEC(ipsec_process_proposal): transform proposal not

supported for identity:

{esp-aes esp-sha-hmac }

This indicates that R1 and R3 cannot find a common transform set to agree upon, so the IPsec SAs

never get established. Let’s see what transform sets R1 and R3 will attempt to negotiate.

R1#show crypto ipsec transform-set

Transform set default: { esp-aes esp-sha-hmac }

will negotiate = { Tunnel, },

Transform set TS: { esp-192-aes esp-sha-hmac }

will negotiate = { Transport, },




Transform set TS: { esp-aes esp-sha-hmac }

will negotiate = { Transport, },

We can already see that the non-default transform set is different, so let’s see if this is the one

actually being used over the DMVPN.

R1#sh run int tun100 | inc ipsec

tunnel protection ipsec profile IPSEC_PROF

R1#sh crypto ipsec profile IPSEC_PROF

IPSEC profile IPSEC_PROF

Security association lifetime: 4608000 kilobytes/3600 seconds

Responder-Only (Y/N): N

PFS (Y/N): N

Transform sets={

TS: { esp-192-aes esp-sha-hmac } ,

}

And let’s check the same on R3.

R3#show crypto ipsec prof IPSEC_PROF




PFS (Y/N): N

Transform sets={

TS: { esp-aes esp-sha-hmac } ,

}

Because R1 already has an established pair of IPsec SAs with R2, then we can assume that R1 has the

correct configuration and R3 has the incorrect configuration. Specifically, R1 is using ESP-AES 192

and R3 is using ESP-AES 128 for encryption in the supported transform set. Let’s fix this on R3.

R3(config)#crypto ipsec transformset TS esp-aes 192 esp-sha-hmac

Soon after we see the EIGRP adjacency come up on R3 to R1 meaning that the crypto must be

working.

R3#



We can also confirm this if we look at the crypto sessions on R1.

R1#show cryp session




Peer: 10.0.13.3 port 500






Peer: 10.0.12.2 port 500




Now let’s run the traceroute in the verification section of the TS scenario (note that it says to run this

command several times to produce the same output).





1 192.168.123.1 5 msec 5 msec 5 msec

2 192.168.123.3 10 msec * 6 msec





1 192.168.123.1 5 msec 2 msec 5 msec

2 192.168.123.3 5 msec * 6 msec





1 192.168.123.1 5 msec 5 msec 5 msec

2 192.168.123.3 6 msec * 6 msec

About 10 seconds passed between each command run, yet the output never changed. Let’s see what

the route looks like on R2 to R3’s loopback.












D 192.168.33.3 [90/102400640] via 192.168.123.1, 00:03:32, Tunnel100

There is no NHO for this route, so maybe there is a problem with the redirect/shortcut configuration.

Let’s check this on each of the routers.

R1#show run int tun100 | inc redirect

no ip redirects

ip nhrp redirect


ip nhrp shortcut


ip nhrp shortcut

That looks fine too so let’s assume that the DMVPN phase 3 configuration is working correctly. The

next thing that should happen is R2 should try and establish an IKE SA with R3 so that they can

communicate directly. Let’s check R2’s crypto sessions.




Session status: DOWN-NEGOTIATING

Peer: 10.0.13.3 port 500

IKEv1 SA: local 10.0.12.2/500 remote 10.0.13.3/500 Inactive





Peer: 10.1.1.1 port 500




R2’s doesn’t seem to be able to establish an IKE SA with R3. Let’s see if the following command will

help us isolate the problem.

R2#show crypto isakmp sa detail

Codes: C - IKE configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal

T - cTCP encapsulation, X - IKE Extended Authentication

psk - Preshared key, rsig - RSA signature

renc - RSA encryption

IPv4 Crypto ISAKMP SA

C-id Local Remote I-VRF Status Encr Hash Auth DH Lifetime

Cap.

1001 10.0.12.2 10.1.1.1 ACTIVE aes sha256 psk 14 23:35:42

Engine-id:Conn-id = SW:1

0 10.0.12.2 10.0.13.3 ACTIVE 0 0

Engine-id:Conn-id = ???

(deleted)

IPv6 Crypto ISAKMP SA

From this output it looks like R2 and R3 are not able to agree upon the parameters that make up the

IKE SA. Let’s check the ISAKMP policy on both routers.

R2#show crypto isakmp policy

Global IKE policy

Protection suite of priority 1

encryption algorithm: AES - Advanced Encryption Standard (256 bit keys).

hash algorithm: Secure Hash Standard 2 (256 bit)

authentication method: Pre-Shared Key

Diffie-Hellman group: #14 (2048 bit)

lifetime: 86400 seconds, no volume limit

R3#show crypto isakmp policy

Global IKE policy

Protection suite of priority 1

encryption algorithm: AES - Advanced Encryption Standard (256 bit keys).

hash algorithm: Secure Hash Standard 2 (256 bit)

authentication method: Pre-Shared Key

Diffie-Hellman group: #14 (2048 bit)

lifetime: 86400 seconds, no volume limit

Looking at this carefully, all of the values seem to be correct so there is no reason why they should

have trouble negotiating the IKE settings. Let’s check out the ISAKMP peer configurations. Let’s

check the ISAKMP key settings.

R2#show crypto isakmp key

Keyring Hostname/Address Preshared Key

default 10.1.1.1 luke123




10.0.12.2 luke123

There we go! R3 has a key configured for R1 and R2 but R2 only has a key configured for R1. Let’s

confirm this in the configuration on R2.

R2#show run | sec isakmp key|keyring

crypto isakmp key luke123 address 10.1.1.1

If there is no key configured for R3, then the IKE SA will never get established. Let’s fix this on R2.

R2(config)#crypto isakmp key luke123 address 10.0.13.3

Let’s clear the crypto sessions on R2 and then confirm that the IPsec SAs get established between R2

and R3.

R2#clear crypto session





Peer: 10.0.13.3 port 500







Peer: 10.1.1.1 port 500




Now IPsec appears to be working between the spokes, so let’s test the traceroute from the original

scenario.





1 192.168.123.3 5 msec * 5 msec

Now everything looks good!


DMVPN and NAT can make for an interesting TS scenario, especially when the hub is behind the NAT

device. When the spoke is behind a NAT device it is not such a big deal because the hub will notice

that the source IP address in NHRP is different from the source IP address in the IP header, and will

infer that NAT is between the two endpoints. There are some things that need to be taken into

consideration with the hub behind NAT, which we will see in the next two scenarios.

First thing to check is to see if any of the spokes have registered with the hub.

R1#show dmvpn






==========================================================================

Which matches what we expect from the original scenario anyway. Starting from R2, let’s check the

NRHP map to the hub.

R2#show ip nhrp

192.168.123.1/32 via 192.168.123.1




Based on the diagram this looks okay, so let’s see if we can ping between the NBMA addresses.

R2#ping 10.0.14.1 source e0/0




.....


Do we have a route to 10.0.14.1?

R2#show ip route 10.0.14.1

% Network not in table

R2#show ip route 0.0.0.0

% Network not in table

This makes sense because R1’s address is a private IP address, and is hidden behind R4 which is

performing NAT. Let’s see how the NAT is setup on R4.

R4#show ip nat statistics

Total active translations: 1 (0 static, 1 dynamic; 0 extended)

Peak translations: 1, occurred 00:11:00 ago

Outside interfaces:

Ethernet0/0, Ethernet0/1, Loopback0

Inside interfaces:

Ethernet0/2

Hits: 0 Misses: 0

CEF Translated packets: 0, CEF Punted packets: 0

Expired translations: 0

Dynamic mappings:

-- Inside Source

Total doors: 0

Appl doors: 0

Normal doors: 0

Queued Packets: 0

So there must be something setup to translate inside local addresses out E0/2 to an inside global

address (or addresses) out either E0/0, E0/1, or Loopback0. We can see if static NAT is configured

using the following command.

R4#show ip nat translations verbose

Pro Inside global Inside local Outside local Outside global

--- 160.4.4.4 10.0.14.1 --- ---

create 00:13:15, use 00:13:15 timeout:0, timing-out,

flags:

static, use_count: 0, entry-id: 2, lc_entries: 0

Based on this, we can see that R4 has a static translation between R1’s E0/0 interface to the address

160.4.4.4. Let’s confirm that this is working by pinging R2 from R1 and seeing what address R2 sees

in the ICMP packet.

R2#debug ip icmp

ICMP packet debugging is on

R1#ping 160.0.24.2 repeat 1



!


Then looking back at the debug output on R2.

R2#

*Dec 7 06:06:54.007: ICMP: echo reply sent, src 160.0.24.2, dst 160.4.4.4,

topology BASE, dscp 0 topoid 0

So NAT appears to be working okay. Therefore we can assume that if we change the NHRP map on

R2 to point to R4’s loopback 0 (160.4.4.4) as the NBMA address, R1 will actually receive the NRHP

packets. Let’s try this out.


R2(config-if)#no ip nhrp map multicast 10.0.14.1


R2(config-if)#ip nhrp map multicast 160.4.4.4


We can actually see that R1 has received the traffic because we see the following message printed to

the console.

R1#




(Tunnel100) is down: retry limit exceeded

We also notice that the adjacency is reset soon after. Let’s check the NHRP cache on R1.

R1#show ip nhrp

192.168.123.2/32


Type: incomplete, Flags: negative

Cache hits: 7

However we have an incomplete NHRP mapping to R2. Let’s confirm that R2 is actually registering

itself to the hub.

R2#show ip nhrp nhs

However we can see that R2 has no NHS configured! Let’s now fix this by setting the NHS to

192.168.123.1.


R2(config-if)#ip nhrp nhs 192.168.123.3

As soon as we configure this we see the following adjacency come up on R2.



Now that it appears R2 is working correctly, let’s look at the current status of the DMVPN from R1’s

perspective.

R1#show dmvpn






==========================================================================

Interface: Tunnel100, IPv4 NHRP Details



----- --------------- --------------- ----- -------- -----

1 160.0.24.2 192.168.123.2 UP 00:01:44 D

R3 is still not registered to R1. Let’s see if there is an NHRP mapping issue on R3 that is preventing

this registration from occurring.

R3#show ip nhrp

192.168.123.1/32 via 192.168.123.1




In a similar fashion to R2, R3 is configured with an NHRP mapping to the inside local address of R1,

which is not reachable by the spokes. Let’s change this on R3.






While we are at it, we may as well see if R3 has no NHS configured like R2.

R3#show ip nhrp nhs

Legend: E=Expecting replies, R=Responding, W=Waiting

Tunnel100:

192.168.123.1 E priority = 0 cluster = 0

R3 is configured correctly here. Soon afterwards, we should see the EIGRP adjacency come up to R1.

R3#



Let’s now test out the verification from the original scenario to make sure we have solved the TS

scenario correctly.





!!!!!


Which we have.


Similar to scenario #4, DMVPN with NAT can be tricky to troubleshoot especially when the hub is

behind a NAT device. In this scenario, IPsec is configured too to make the scenario even harder. This

should test your understanding of how crypto is handled with NAT and different IPsec modes. The

last debug command from the original scenario actually provides the biggest clue to one of the

problems.

Let’s first see if any of the spoke routers have registered with the hub.

R1#show dmvpn






Nothing has registered to R1. Let’s check the NHRP mappings and the NHS setting on R2 and R3.

R2#show ip nhrp

192.168.123.1/32 via 192.168.123.1




R2#show ip nhrp nhs


Tunnel100:


R3#show ip nhrp

192.168.123.1/32 via 192.168.123.1




R3#show ip nhrp nhs


Tunnel100:


This all looks fine. If we take a step back let’s see if IKE or IPsec SAs are being established on R1.


Is there a problem with NAT on R4?

R4#show ip nat translations verbose


esp 160.4.4.4:0 10.0.14.1:0 --- ---


flags:

extended, extendable, static, use_count: 0, entry-id: 1, lc_entries: 0

udp 160.4.4.4:500 10.0.14.1:500 --- ---


flags:

extended, extendable, static, use_count: 0, entry-id: 2, lc_entries: 0

The NAT entries look fine as they allow ISAKMP (UDP/500) and ESP traffic to pass between R1 and

the outside world. Let’s run a debug for ISAKMP on R2 to see if we can isolate any problems with the

ISAKMP negotiation.

R2#debug crypto isakmp

Crypto ISAKMP debugging is on

R2#ping 192.168.123.1 repeat 1



*Dec 7 07:36:29.419: ISAKMP:(0): SA request profile is (NULL)

*Dec 7 07:36:29.419: ISAKMP: Created a peer struct for 160.4.4.4, peer port 500

*Dec 7 07:36:29.419: ISAKMP: New peer created peer = 0xB2C81C38 peer_handle =

0x80000014

*Dec 7 07:36:29.419: ISAKMP: Locking peer struct 0xB2C81C38, refcount 1 for

isakmp_initiator

*Dec 7 07:36:29.419: ISAKMP: local port 500, remote port 500

*Dec 7 07:36:29.419: ISAKMP: set new node 0 to QM_IDLE

*Dec 7 07:36:29.419: ISAKMP:(0):insert sa successfully sa = B29F17D8

*Dec 7 07:36:29.419: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.

*Dec 7 07:36:29.419: ISAKMP:(0):No pre-shared key with 160.4.4.4!

*Dec 7 07:36:29.419: ISAKMP:(0): No Cert or pre-shared address key.

*Dec 7 07:36:29.419: ISAKMP:(0): construct_initial_message: Can not start Main

mode

*Dec 7 07:36:29.419: ISAKMP: Unlocking peer struct 0xB2C81C38 for

isadb_unlock_peer_delete_sa(), count 0

*Dec 7 07:36:29.419: ISAKMP: Deleting peer node by peer_reap for 160.4.4.4:

B2C81C38

*Dec 7 07:36:29.419: ISAKMP:(0):purging SA., sa=B29F17D8, delme=B29F17D8

*Dec 7 07:36:29.419: ISAKMP:(0):purging node -1924640252

*Dec 7 07:36:29.419: ISAKMP: Error while processing SA request: Failed to

initialize SA

*Dec 7 07:36:29.419: ISAKMP: Error while processing KMI message 0, error 2..


Excellent! This debug has told us that there is no pre-shared key with 160.4.4.4 (the inside global

address of the hub). Let’s check this setting on R2.




Okay so we have configured the key to the inside local address rather than the inside global. We will

fix this as follows.

R2(config)#no crypto isakmp key luke123 address 10.0.14.1

R2(config)#crypto isakmp key luke123 address 0.0.0.0 0.0.0.0

It is probable that we are going to need spoke to spoke communications to occur, so we simply use

the key “luke123” for all ISAKMP peers. Let’s now look on R1 to see if the crypto session has come

up.



Interface: Ethernet0/0

Session status: DOWN-NEGOTIATING

Peer: 160.0.24.2 port 500

IKEv1 SA: local 10.0.14.1/500 remote 160.0.24.2/500 Inactive

It’s still not coming up so what else could be wrong with IPsec through NAT? This is where the hint

from the original scenario comes into play. Let’s run a debug on R1 to show what is wrong.

R1#debug crypto isakmp


R1#

*Dec 7 07:43:36.496: ISAKMP (0): received packet from 160.0.24.2 dport 500 sport

500 Global (N) NEW SA


*Dec 7 07:43:36.496: ISAKMP: New peer created peer = 0xB2C2FE70 peer_handle =

0x80000003

<snip>

*Dec 7 07:43:36.527: ISAKMP (1002): NAT found, both nodes inside NAT

*Dec 7 07:43:36.527: ISAKMP:received payload type 20

*Dec 7 07:43:36.527: ISAKMP (1002): My hash no match - this node inside NAT

R1#

There is a lot of output when running the above command, but if you know have ever setup IPsec

through NAT you will have seen this problem before. If it is still not clicking than the debug output

on R2 should highlight the exact problem for you.

R2#debug cry isak


R2#

*Dec 7 07:48:13.469: ISAKMP:(1004):purging node 1258689276

*Dec 7 07:48:13.469: ISAKMP:(1004):purging node -1579434794

*Dec 7 07:48:14.030: ISAKMP:(0): SA request profile is (NULL)


*Dec 7 07:48:14.030: ISAKMP: New peer created peer = 0xB2C81C38 peer_handle =

0x8000001E

<snip>

*Dec 7 07:48:14.064: ISAKMP (1005): His hash no match - this node outside NAT

*Dec 7 07:48:14.064: ISAKMP:received payload type 20

*Dec 7 07:48:14.064: ISAKMP (1005): His hash no match - this node outside NAT

<snip>

*Dec 7 07:48:14.064: ISAKMP:(1005): sending packet to 160.4.4.4 my_port 4500

peer_port 4500 (I) MM_KEY_EXCH

*Dec 7 07:48:14.064: ISAKMP:(1005):Sending an IKE IPv4 Packet.

*Dec 7 07:48:14.064: ISAKMP:(1005):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE

*Dec 7 07:48:14.064: ISAKMP:(1005):Old State = IKE_I_MM4 New State = IKE_I_MM5

R2#

*Dec 7 07:48:23.470: ISAKMP:(1004):purging SA., sa=B1053CE8, delme=B1053CE8

*Dec 7 07:48:24.055: ISAKMP (1005): received packet from 160.4.4.4 dport 500 sport

500 Global (I) MM_KEY_EXCH

*Dec 7 07:48:24.055: ISAKMP:(1005): phase 1 packet is a duplicate of a previous

packet.

*Dec 7 07:48:24.055: ISAKMP:(1005): retransmitting due to retransmit phase 1

R2#

*Dec 7 07:48:24.560: ISAKMP:(1005): retransmitting phase 1 MM_KEY_EXCH...

*Dec 7 07:48:24.560: ISAKMP (1005): incrementing error counter on sa, attempt 1 of

5: retransmit phase 1

*Dec 7 07:48:24.560: ISAKMP:(1005): retransmitting phase 1 MM_KEY_EXCH

*Dec 7 07:48:24.560: ISAKMP:(1005): sending packet to 160.4.4.4 my_port 4500

peer_port 4500 (I) MM_KEY_EXCH

*Dec 7 07:48:24.560: ISAKMP:(1005):Sending an IKE IPv4 Packet.

When IPsec is used with NAT, NAT-T is used. NAT-T, discussed in RFC 3947 and RFC 3948, provides a

method to detect a NAT device between the IPsec endpoints and also a way to get around PAT (as

IPsec is not associated with a transport layer port) by encapsulating the ESP tunnel mode payload

with UDP or using transport mode.

So given this new piece of information, we should check whether R4 has a NAT rule for UDP port

4500 (NAT-T).

R4#show ip nat translations


esp 160.4.4.4:0 10.0.14.1:0 --- ---

udp 160.4.4.4:500 10.0.14.1:500 --- ---

Which it does not. Let’s add a rule to allow this translation to occur.

R4(config)#ip nat inside source static udp 10.0.14.1 4500 interface Loopback0 4500

Let’s now see if this has fixed the crypto problem between R1 and R2.



Interface: Ethernet0/0

Session status: UP-IDLE

Peer: 160.0.24.2 port 4500


Well at least we have the IKE SA now so that’s looking better already. Let’s now see if there is a

problem with the IPsec transform set between R1 and R2.











The problem has already showed itself above, but we will confirm it with a debug on either R1 or R2.

R1#debug crypto ipsec

Crypto IPSEC debugging is on

R1#

*Dec 7 08:04:39.389: IPSEC(key_engine): got a queue event with 1 KMI message(s)

R1#

*Dec 7 08:04:58.613: IPSEC(key_engine): got a queue event with 1 KMI message(s)

*Dec 7 08:04:58.619: IPSEC(validate_proposal_request): proposal part #1

*Dec 7 08:04:58.619: IPSEC(validate_proposal_request): proposal part #1,

(key eng. msg.) INBOUND local= 10.0.14.1:0, remote= 160.0.24.2:0,

local_proxy= 160.4.4.4/255.255.255.255/47/0,

remote_proxy= 160.0.24.2/255.255.255.255/47/0,

protocol= ESP, transform= NONE (Tunnel-UDP),

lifedur= 0s and 0kb,

spi= 0x0(0), conn_id= 0, keysize= 192, flags= 0x0

*Dec 7 08:04:58.619: map_db_find_best did not find matching map

*Dec 7 08:04:58.619: IPSEC(ipsec_process_proposal): proxy identities not supported

If you read this link you will see that one of the pre-requisites for NAT-T to work in a DMVPN is that

transport mode must be used. Let’s change the transport mode on both R1 and R2 to see if this fixes

our problem.

R1(config)#crypto ipsec transform-set TS esp-aes 192 esp-sha-hmac

R1(cfg-crypto-trans)#mode transport



Shortly after you should see the EIGRP adjacency come up between R1 and R2.

R2#



R2#


(Tunnel100) is down: retry limit exceeded

However the adjacency is flapping, which as we discovered in an earlier task, usually means that

there is a problem with multicast mapping. Let’s check it on R1.

R2#show ip nhrp multicast

I/F NBMA address

Tunnel100 10.0.14.1 Flags: static

We can see the problem straight away. R2 has an NHRP map for multicast to the inside local address

for R1, not the inside global address. Let’s fix this as follows.




Let’s see if this has fixed the flapping EIGRP issue.

R2#show ip eigrp nei

EIGRP-IPv4 VR(CCIE) Address-Family Neighbors for AS(123)

H Address Interface Hold Uptime SRTT RTO Q Seq

(sec) (ms) Cnt Num

0 192.168.123.1 Tu100 10 00:01:05 2 1470 0 3

http://www.cisco.com/c/en/us/td/docs/ios/sec_secure_connectivity/configuration/guide/15_0/sec_secure_connectivity_15_0_book/sec_DMVPN.html#wp1122466

The queue count here is zero which is good. Let’s see if R1 is receiving routes now.











D 192.168.22.0/24 [90/76800640] via 192.168.123.2, 00:02:05, Tunnel100

This is looking much better! Now we must figure out what is wrong with R3. From earlier we

confirmed that the NHRP mappings and NHS settings looked okay. Let’s see if we also have tunnel

mode ESP configured rather than transport mode.






Let’s change this to transport mode now.



But nothing seems to happen. Has any crypto session state been created on R3?


If the DMVPN tunnel is up, then the endpoints should definitely be trying to negotiate an IKE SA.

Let’s check to see if the tunnel is up.

R3#show ip int Tun100 | inc line

Tunnel100 is up, line protocol is up

Has an IPsec crypto profile been created for use on the tunnel?

R3#show crypto ipsec profile




PFS (Y/N): N

Transform sets={

TS: { esp-192-aes esp-sha-hmac } ,

}

IPSEC profile default



PFS (Y/N): N

Transform sets={

default: { esp-aes esp-sha-hmac } ,

}

This looks good, but has it been enabled on the tunnel?

R3#sh run int tun100



!

interface Tunnel100

description DMVPN

ip address 192.168.123.3 255.255.255.0

no ip redirects



ip nhrp map 192.168.123.1 160.4.4.4


ip nhrp nhs 192.168.123.1

ip nhrp redirect



end

And there we see the problem: the crypto profile has not been configured under the tunnel

interface. Let’s fix this.


R3(config-if)#tunnel protection ipsec profile IPSEC_PROF

Straight away we see that this has fixed the problem because the ISAKMP process starts and the

EIGRP adjacency comes up. This is evident from the following messages.

*Dec 7 08:30:51.366: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON

R3#



R3#

Now as a final test, let’s see if R2 and R3 can ping between their loopbacks.





!!!!!


dynamic multipoint vpn (dmvpn) troubleshooting · pdf filescenario #1 solutions when we run...

Documents