7200 with VTI ipsec issues

jonathan johnson · ‎05-21-2013

Hi,

We have a Cisco 7204 G1 running c7200-advipservicesk9-mz.122-33.SRE7.bin and we're having a lot of difficulties getting a VTI working to a Cisco 2921 with adv. security.

I've ruled out that the 2921 is at fault by successfully establishing a VTI to another 2921 and a 7200 running a different IOS release.

We see the tunnel come up, but when I sent a ping from the 2921 to the 7204 there isn't a reply.

When I look at the results on the 7204 from a 'sh crypto engin connection active', I see the decrypt counters increase, but I don't see the Encrypt counters increase as it's trying to reply to the ping. I'm not sure if this is because there is an issue with the encryption or wheather there might be a more fundemental issue with the router not replying to the pings.

I've tried the following IOS releases (c7200-advipservicesk9-mz.122-33.SRE7 & c7200-advipservicesk9-mz.122-33.SRE6) and they all behave the same way - this makes me think it might be a config issue rather than and IOS bug which is what I first thought.

c7200-advipservicesk9-mz.122-33.SRE7.bin

sh crypto engine connections active

Crypto Engine Connections

ID Interface Type Algorithm Encrypt Decrypt IP-Address

1 Tu10 IPsec 3DES+SHA 0 31 10.5.5.1

2 Tu10 IPsec 3DES+SHA 19 0 10.5.5.1

1001 Tu10 IKE SHA+3DES 0 0 10.5.5.1

Here is a copy of my config on the 7204 - the other end (Cisco 2921) is configured in the same way

crypto isakmp policy 1

encr 3des

authentication pre-share

group 2

crypto isakmp key xxxxxxxx address 192.168.5.1

!

crypto ipsec transform-set 3DESSHA esp-3des esp-sha-hmac

!

crypto ipsec profile IPSEC-VPN

set transform-set 3DESSHA

interface Loopback10

ip address 10.5.5.1 255.255.255.255

!

interface Tunnel10

ip address 10.10.10.1 255.255.255.252

tunnel source Loopback10

tunnel mode ipsec ipv4

tunnel destination 192.168.5.1

tunnel protection ipsec profile IPSEC-VPN

Any assistance would be greatly appreciated.

Thanks,

Jonathan.

jonathan johnson · ‎05-21-2013

I forgot to mention that I'm sending the ping traffic from router to router - pinging the remote end of the tunnel.

Richard Burts · ‎05-22-2013

Jonathan

I do not see an obvious issue in the part of the config that you posted. So we need to dig a bit deeper in looking for the cause of the problem. Can you post the following outputs for the 7200:

- show ip interface brief

- show crypto ipsec sa

- traceroute 192.168.5.1

HTH

Rick

HTH

Rick

jonathan johnson · ‎05-22-2013

Hi Rick,

Thanks for your reply.

Here's output from the commands you requested.

sh ip interface brief

Interface IP-Address OK? Method Status Protocol

GigabitEthernet0/1 unassigned YES NVRAM up up

GigabitEthernet0/1.43 x.x.x.x YES NVRAM up up

GigabitEthernet0/2 unassigned YES NVRAM up up

GigabitEthernet0/2.42 x.x.x.x YES NVRAM up up

GigabitEthernet0/3 unassigned YES NVRAM administratively down down

Loopback0 x.x.x.x YES NVRAM up up

Loopback10 10.5.5.1 YES NVRAM up up

Tunnel10 10.10.10.1 YES NVRAM up up

traceroute 192.168.5.1

Type escape sequence to abort.

Tracing the route to 1-38-17-10.static.test.com (192.168.5.1)

1 62-60-17-10.static.test.com (10.17.60.62) [MPLS: Label 660 Exp 0] 16 msec 12 msec 12 msec

2 81-60-17-10.static.test.com (10.17.60.81) [MPLS: Label 396 Exp 0] 12 msec 12 msec 12 msec

3 81-63-17-10.static.test.com (10.17.63.81) [MPLS: Label 694 Exp 0] 12 msec 12 msec 12 msec

4 129-63-17-10.static.test.com (10.17.63.129) 12 msec 12 msec 12 msec

5 34-37-17-10.static.test.com (10.17.37.34) 28 msec * 24 msec

#sh crypto ipsec sa

interface: Tunnel10

Crypto map tag: Tunnel10-head-0, local addr 10.5.5.1

protected vrf: (none)

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

current_peer 192.168.5.1 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 66, #pkts encrypt: 66, #pkts digest: 66

#pkts decaps: 116, #pkts decrypt: 116, #pkts verify: 116

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 0

local crypto endpt.: 10.5.5.1 , remote crypto endpt.: 192.168.5.1

path mtu 1514, ip mtu 1514

current outbound spi: 0x417F00DA(1098842330)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0xA0613593(2690725267)

transform: esp-3des esp-sha-hmac ,

in use settings ={Tunnel, }

conn id: 37, flow_id: 37, sibling flags 80000040, crypto map: Tunnel10-head-0

sa timing: remaining key lifetime (k/sec): (4469938/1859)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x417F00DA(1098842330)

transform: esp-3des esp-sha-hmac ,

in use settings ={Tunnel, }

conn id: 38, flow_id: 38, sibling flags 80000040, crypto map: Tunnel10-head-0

sa timing: remaining key lifetime (k/sec): (4469938/1859)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

Thanks,

Jonathan

Richard Burts · ‎05-22-2013

Jonathan

Thank you for the outputs that I requested. I find them encouraging. If these are from the 7200 it might be helpful to see the same outputs from the 2921.

What I am seeing in these outputs seems to confirm that the tunnel is working ok. In particular I note these points:

- one requirement for VTI tunnels is that there is IP connectivity between the tunnel source and the tunnel destination. The traceroute does seem to demonstrate that this connectivity does exist.

- one of the requirements for the VTI tunnel to come up an up/up state is that the crypto negotiation was successful. The fact that the tunnel does show as up/up confirms that there is crypto negotiation.

- in the IPSec Security Association the counters for encrypted and decrypted are both non-zero and this indicates that there is two way traffic flowing through the tunnel.

#pkts encaps: 66, #pkts encrypt: 66, #pkts digest: 66

#pkts decaps: 116, #pkts decrypt: 116, #pkts verify: 116

I would like to request one more output from the 7200. Please post the output of show ip route 10.10.10.2

If the tunnel seems to be working but the ping is not working them there must be some other cause of the problem other than something wrong with the tunnel. Perhaps you can post the configuration of the 2921? And while I am asking for things perhaps also the same outputs from the 2921 that I requested from the 7200?

HTH

Rick

HTH

Rick

jonathan johnson · ‎05-22-2013

Hi Rick,

Here is the output you requested. Something to note, when the IPSec config from the tunnel is removed (so it becomes a standard gre tunnel) traffic goes through without a problem.

##################################################################################

Show ip route from the 7200

sh ip route 10.10.10.2

Routing entry for 10.10.10.0/30

Known via "connected", distance 0, metric 0 (connected, via interface)

Redistributing via ospf 10

Advertised by ospf 10 subnets

Routing Descriptor Blocks:

* directly connected, via Tunnel10

Route metric is 0, traffic share count is 1

#####################################################################################

version 15.1

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname 2921

!

boot-start-marker

boot system flash:c2900-universalk9-mz.SPA.151-4.M4.bin

boot system flash:c2900-universalk9-mz.SPA.151-4.M3.bin

boot-end-marker

!

logging buffered 51200 warnings

!

no aaa new-model

!

no ipv6 cef

ipv6 spd queue min-threshold 62

ipv6 spd queue max-threshold 63

no ip source-route

ip cef

!

no ip domain lookup

!

multilink bundle-name authenticated

!

crypto isakmp policy 1

encr 3des

authentication pre-share

group 2

crypto isakmp key xxxxxxx address 10.5.5.1

!

crypto ipsec transform-set 3DESSHA esp-3des esp-sha-hmac

!

crypto ipsec profile IPSEC-VPN

set transform-set 3DESSHA

!

interface Loopback0

description Source interface for tunnel to backoffice

ip address 192.168.5.1 255.255.255.255

!

interface Tunnel10

description tunel to 7200

ip address 10.10.10.2 255.255.255.252

tunnel source loopback0

tunnel mode ipsec ipv4

tunnel destination 10.5.5.1

tunnel protection ipsec profile IPSEC-VPN

!

interface GigabitEthernet0/0

no ip address

shutdown

duplex full

speed 1000

!

interface GigabitEthernet0/1

description Fiber to upstream

ip address 10.17.37.34 255.255.255.252

duplex auto

speed auto

!

interface GigabitEthernet0/2

description LAN network

ip address 10.17.37.1 255.255.255.224

duplex auto

speed auto

!

no ip http server

no ip http secure-server

!

ip route 0.0.0.0 0.0.0.0 10.17.37.33 250

###########################################################

sh ip int brief

Interface IP-Address OK? Method Status Protocol

Embedded-Service-Engine0/0 unassigned YES NVRAM administratively down down

GigabitEthernet0/0 unassigned YES manual administratively down down

GigabitEthernet0/1 10.17.37.34 YES manual up up

GigabitEthernet0/2 10.17.37.1 YES NVRAM up up

Loopback0 192.168.5.1 YES NVRAM up up

Tunnel10 10.10.10.2 YES manual up up

##########################################################

show crypto ipsec sa

interface: Tunnel10

Crypto map tag: Tunnel10-head-0, local addr 192.168.5.1

protected vrf: (none)

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

current_peer 10.5.5.1 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 136, #pkts encrypt: 136, #pkts digest: 136

#pkts decaps: 86, #pkts decrypt: 86, #pkts verify: 86

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 0

local crypto endpt.: 192.168.5.1, remote crypto endpt.: 10.5.5.1

path mtu 1514, ip mtu 1514, ip mtu idb Loopback0

current outbound spi: 0x965C2BDA(2522622938)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0x65D51986(1708464518)

transform: esp-3des esp-sha-hmac ,

in use settings ={Tunnel, }

conn id: 2087, flow_id: Onboard VPN:87, sibling_flags 80000046, crypto map: Tunnel10-head-0

sa timing: remaining key lifetime (k/sec): (4554753/1417)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x965C2BDA(2522622938)

transform: esp-3des esp-sha-hmac ,

in use settings ={Tunnel, }

conn id: 2088, flow_id: Onboard VPN:88, sibling_flags 80000046, crypto map: Tunnel10-head-0

sa timing: remaining key lifetime (k/sec): (4554753/1417)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

#########################################################

traceroute 10.5.5.1

Type escape sequence to abort.

Tracing the route to 10.5.5.1

1 10.17.37.33 12 msec 12 msec 12 msec

2 10.17.63.141 28 msec 24 msec 24 msec

3 10.17.63.86 24 msec 24 msec 24 msec

4 10.17.60.82 24 msec 28 msec 24 msec

5 10.17.60.49 24 msec * 24 msec

Many thanks,

Jonathan

Richard Burts · ‎05-22-2013

Jonathan

Thanks for these outputs. They seem to confirm that the tunnel is working ok. Unfortunately they do not suggest what the problem is that is preventing ping to the other end of the tunnel.

So I will suggest an additional test. Please enable debug ip icmp on both routers. then ping from one router to the other. capture the debug output and post it. I note that the 2921 sets the logging level for logging buffered to warning and I wonder if the 7200 does this also. Either you need to have active sessions to the router (and verify that the logging level for monitor is set to debug) or you need to change the configured logging buffered level to debug.

It might be a better test to do ping from each of the routers and to mark in the debug output which is which.

HTH

Rick

HTH

Rick

jonathan johnson · ‎05-22-2013

Hi Richard,

Here the output from the 2921 when I send 5 pings from the 7200

May 22 23:25:36.987: ICMP: echo reply sent, src 10.10.10.2, dst 10.10.10.1, topology BASE, dscp 0 topoid0

May 22 23:25:38.987: ICMP: echo reply sent, src 10.10.10.2, dst 10.10.10.1, topology BASE, dscp 0 topoid0

May 22 23:25:40.987: ICMP: echo reply sent, src 10.10.10.2, dst 10.10.10.1, topology BASE, dscp 0 topoid0

May 22 23:25:42.987: ICMP: echo reply sent, src 10.10.10.2, dst 10.10.10.1, topology BASE, dscp 0 topoid0

May 22 23:25:44.987: ICMP: echo reply sent, src 10.10.10.2, dst 10.10.10.1, topology BASE, dscp 0 topoid0

When I do the same from the 2921 back to the 7200 I get no debug output at all....

But, I'm quite sure they arrive and are 'seen' by the router at some level because the Decrypt counters increase while pings are being sent.

ID Interface Type Algorithm Encrypt Decrypt IP-Address

45 Tu10 IPsec 3DES+SHA 0 73 10.5.5.1

46 Tu10 IPsec 3DES+SHA 5 0 10.5.5.1

1003 Tu10 IKE SHA+3DES 0 0 10.5.5.1

72001#sh crypto engine connections active

Crypto Engine Connections

ID Interface Type Algorithm Encrypt Decrypt IP-Address

45 Tu10 IPsec 3DES+SHA 0 73 10.5.5.1

46 Tu10 IPsec 3DES+SHA 5 0 10.5.5.1

1003 Tu10 IKE SHA+3DES 0 0 10.5.5.1

7200#sh crypto engine connections active

Crypto Engine Connections

ID Interface Type Algorithm Encrypt Decrypt IP-Address

45 Tu10 IPsec 3DES+SHA 0 74 10.5.5.1

46 Tu10 IPsec 3DES+SHA 5 0 10.5.5.1

1003 Tu10 IKE SHA+3DES 0 0 10.5.5.1

Thanks,

Jonathan

Richard Burts · ‎05-22-2013

Jonathan

Can you verify that debug is enabled on the 7200 and that logging is enabled at the level of debug and try the ping from the 2921 again? and post whether there is any debug output?

HTH

Rick

HTH

Rick

jonathan johnson · ‎05-22-2013

Hi Rick,

Yes, logging is enabled on the 7200, I received some icmp debug from our monitoring system that is sending pings to the 7200 - but nothing from the 2921.

Thanks,

Jonathan

Richard Burts · ‎05-22-2013

Jonathan

That is quite puzzling.

Please post the output from the 2921 of show ip route 10.10.10.1

My second suggestion is to configure some other address on the 7200 (perhaps some other physical interface or perhaps another loopback interface) and then on the 2921 configure a static route that says you can get to the 7200 other address via the next hop through the tunnel. Then ping and traceroute to that address from the 2921.

My third suggestion is to configure a routing protocol (perhaps EIGRP) and configure it to run over the tunnel. It does not necessarily need to advertise other subnets yet, but I would like to see whether it is able to establish neighbor relationships over the tunnel.

HTH

Rick

HTH

Rick

jonathan johnson · ‎05-22-2013

Hi Rick,

I agree it's very puzzling....

Here is the output of the sh ip route command from the 2921

sh ip route 10.10.10.1

Routing entry for 10.10.10.0/30

Known via "connected", distance 0, metric 0 (connected, via interface)

Routing Descriptor Blocks:

* directly connected, via Tunnel10

Route metric is 0, traffic share count is 1

When I add the static route I don't get a reply to pings and a traceroute gives the following output:

Tracing the route to 10.17.60.252

VRF info: (vrf in name/id, vrf out name/id)

1 * * *

2 * * *

3 * * *

4 * *

I configured ospf on both sides. The 2921 tied to bring it up, but stayed in the 'INIT' state.

The 7200 sends 'hello' packets, and I see these get to the 2921, but, the 7200 never gets the hello messages from the 2921. This is the same behaviour that we see with the icmp debug.

Thanks,

Jonathan

Richard Burts · ‎05-22-2013

Jonathan

This feels very much like there is some routing issue involved, and most likely on the 7200. You have given us some (but not all) of the config of the 2921 and very little of the config of the 7200. I wonder if there is some clue to the issue in the parts of the config that you have not yet posted. Can you give us some more of the config?

Perhaps the output of show ip route from the 7200 might give us some clue. Also perhaps the output of show ip protocol.

I am also wondering about what is running on the 7204. Perhaps you can post the output of show version?

Most of the 7200 that I have experience with were running 12.3 or 12.4 versions. You indicate that it is running c7200-advipservicesk9-mz.122-33.SRE7.bin. 12.2(33)SRE7 seems unusual to me. I will do some looking about that. If there is anything that you can give us about the code that you are running might be helpful.

HTH

Rick

HTH

Rick

Richard Burts · ‎05-22-2013

Jonathan

I am finding it difficult to find good information. But what I am finding suggests that what you have is the ubr7204, which is a router for cable modem termination systems and not the general purpose 7204 that i was assuming that you were using. What I am seeing suggests that VTI is not supported in that platform. In fact I am surprised that the commands to configure it were accepted, if I am correct about your platform. So I am waiting for the show version and any other information that you can provide about the details of the platform that you are using.

HTH

Rick

HTH

Rick

Richard Burts · ‎05-22-2013

Jonathan

I just found a reference in the Cisco Feature Navigator that says that VTI is supported in the version of code that you are running. So now I am quite confused. But I am leaning toward it being buggy behavior in the IOS.

HTH

Rick

HTH

Rick