IPsec over GRE - unable to ping remote end

[kamrannaseem1]

Hello All,

I am having issue with the IPsec over GRE, I am unable to ping the  remote end and they are unable to ping me.  When the remote end ping us we can see the packet decaps going up but when we ping them we don not see increase in packet encaps.

Please see below the config and crypto session details:

========================
========================

Routitng Table

C 10.7.64.32/30 is directly connected, Loopback2751
L 10.7.64.33/32 is directly connected, Loopback2751
S 10.100.0.0/28 is directly connected, Tunnel1751
169.250.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 169.250.20.20/30 is directly connected, Tunnel1751
L 169.250.20.21/32 is directly connected, Tunnel1751

========================
========================

interface Tunnel1751
ip vrf forwarding SimulationSystems-106816
ip address 169.250.20.21 255.255.255.252
ip virtual-reassembly in
ip tcp adjust-mss 1379
tunnel source x.x.x.x
tunnel mode ipsec ipv4
tunnel destination x.x.x.x
tunnel protection ipsec profile SimulationSystems-106816-A

========================
========================

interface Loopback2751
ip vrf forwarding SimulationSystems-106816
ip address 10.7.64.33 255.255.255.252

========================
========================

Cisco_router#sh crypto session ivrf SimulationSystems-106816 detail
Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect

Interface: Tunnel1751
Profile: SimulationSystems-106816-profile-A
Uptime: 01:19:27
Session status: UP-ACTIVE
Peer: x.x.x.x port 4500 fvrf: (none) ivrf: SimulationSystems-106816
Phase1_id: x.x.x.x
Desc: (none)
Session ID: 648164
IKEv2 SA: local x.x.x.x/4500 remote x.x.x.x/4500 Active
Capabilities:N connid:2 lifetime:06:40:33
IPSEC FLOW: permit 47 10.7.64.32/255.255.255.252 10.100.0.0/255.255.255.240
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 24515 drop 0 life (KB/Sec) 4240071/23 hours, 36 mins
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4240071/23 hours, 36 mins

Interface: Tunnel1751
Session status: DOWN
Peer: 52.213.20.116 port 500 fvrf: (none) ivrf: SimulationSystems-106816
Desc: (none)
Phase1_id: (none)
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 0/0

[agairola]

Hi,

We can try to debug the datapath seating on the router by using “debug ip packet” but if we do not with condition then it can be very CPU intensive.

Flow below document to configure debug ip packet:

https://goo.gl/HftQao

We can also enable CEF debugging to check if cef is not miss-behaving – “debug ip cef drop”

./Adesh

[kamrannaseem1]

Thank you for the response Adesh.

So If i create an access-list to debug IP packet what should be my source and destination IP ?
Should it be the tunnel IP addresses ?

[agairola]

On the second look, something does look fishy in the IPsec output:

interface Tunnel1751
ip vrf forwarding SimulationSystems-106816
ip address 169.250.20.21 255.255.255.252
ip virtual-reassembly in
ip tcp adjust-mss 1379
tunnel source x.x.x.x
tunnel mode ipsec ipv4 >>>>>>>>>>> The mode is IPsec IPv4 which should have 0.0.0.0/0.0.0.0 IPsec SA
tunnel destination x.x.x.x
tunnel protection ipsec profile SimulationSystems-106816-A

But as per below command its permitting GRE with some ACL:

Cisco_router#sh crypto session ivrf SimulationSystems-106816 detail
Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect

Interface: Tunnel1751
Profile: SimulationSystems-106816-profile-A
Uptime: 01:19:27
Session status: UP-ACTIVE
Peer: x.x.x.x port 4500 fvrf: (none) ivrf: SimulationSystems-106816
Phase1_id: x.x.x.x
Desc: (none)
Session ID: 648164
IKEv2 SA: local x.x.x.x/4500 remote x.x.x.x/4500 Active
Capabilities:N connid:2 lifetime:06:40:33
IPSEC FLOW: permit 47 10.7.64.32/255.255.255.252 10.100.0.0/255.255.255.240 >>>>>>>>>>>>>> This output
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 24515 drop 0 life (KB/Sec) 4240071/23 hours, 36 mins
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4240071/23 hours, 36 mins

Can you provide the full "show tech"or "show run" and below output:

# show crypto isa sa

# show crypto ikev2 sa [if control is on IKEv2]

# show crypto ipsec sa interface Tunnel1751

[kamrannaseem1]

Cisco#show crypto ipsec sa interface Tunnel1751

interface: Tunnel1751
Crypto map tag: Tunnel1751-head-0, local addr 95.x.x.x

protected vrf: SimulationSystems-106816
local ident (addr/mask/prot/port): (10.7.64.32/255.255.255.252/47/0)
remote ident (addr/mask/prot/port): (10.100.0.0/255.255.255.240/47/0)
current_peer 52.x.x.x port 4500
PERMIT, flags={}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 1566, #pkts decrypt: 1566, #pkts verify: 1566
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 95.x.x.x, remote crypto endpt.: 52.x.x.x
plaintext mtu 1406, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0xC35448EC(3277080812)
PFS (Y/N): Y, DH group: group14

inbound esp sas:
spi: 0x982D7C3F(2553117759)
transform: esp-256-aes esp-sha512-hmac ,
in use settings ={Tunnel UDP-Encaps, }
conn id: 7815, flow_id: Onboard VPN:7815, sibling_flags 80000040, crypto map: Tunnel1751-head-0
sa timing: remaining key lifetime (k/sec): (4294391/86136)
IV size: 16 bytes
replay detection support: Y replay window size: 128
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xC35448EC(3277080812)
transform: esp-256-aes esp-sha512-hmac ,
in use settings ={Tunnel UDP-Encaps, }
conn id: 7816, flow_id: Onboard VPN:7816, sibling_flags 80000040, crypto map: Tunnel1751-head-0
sa timing: remaining key lifetime (k/sec): (4294442/86136)
IV size: 16 bytes
replay detection support: Y replay window size: 128
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

protected vrf: SimulationSystems-106816
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
current_peer 52.x.x.x port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 95.x.x.x, remote crypto endpt.: 52.x.x.x
plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

=============

Cisco#sh crypto ikev2 sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
 95.x.x.x/4500   52.x.x.x/4500 none/SimulationSys READY
Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:14, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/11432 sec

==============

Cisco#sh crypto isakmp sa

nothing showing in the output

[kamrannaseem1]

Hi Adesh,

When I run the below command I see Interface: Tunnel1751 shown twice one is showing UP-ACTIVE and the other one is showing as DOWN.  Please see below the output.

Cisco#sh crypto session ivrf SimulationSystems-106816 detail
Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect

Interface: Tunnel1751
Profile: SimulationSystems-106816-profile-A
Uptime: 04:10:25
Session status: UP-ACTIVE
Peer: 52.x.x.x port 4500 fvrf: (none) ivrf: SimulationSystems-106816
Phase1_id: 52.x.x.x
Desc: (none)
Session ID: 649529
IKEv2 SA: local 95.x.x.x/4500 remote 52.x.x.x/4500 Active
Capabilities:N connid:5 lifetime:03:49:35
IPSEC FLOW: permit 47 10.7.64.32/255.255.255.252 10.100.0.0/255.255.255.240
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 6477 drop 0 life (KB/Sec) 4323527/23 hours, 28 mins
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4323894/23 hours, 28 mins

Interface: Tunnel1751
Session status: DOWN
Peer: 52.x.x.x.x port 500 fvrf: (none) ivrf: SimulationSystems-106816
Desc: (none)
Phase1_id: (none)
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 0/0

[agairola · Accepted solution]

Hi,

I think I have a fair idea about the issue that you would be facing here.

Short story, you may have to match the way remote VPN peer has created IPsec-over-GRE. You change accordingly or they have to change.

Long story, here is a reason for what I said:

** Your side is configured to do pure IPsec (its VTI based) not IPsecoverGRE, you can check it with below config and output:

interface Tunnel1751

ip vrf forwarding SimulationSystems-106816

ip address 169.250.20.21 255.255.255.252

ip virtual-reassembly in

ip tcp adjust-mss 1379

tunnel source x.x.x.x

tunnel mode ipsec ipv4 >>>>>>>>>>> The mode is pure IPsec, to enable GRE it should be “tunnel mode gre”

tunnel destination x.x.x.x

tunnel protection ipsec profile SimulationSystems-106816-A

protected vrf: SimulationSystems-106816

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0) >>>> no GRE (47), its pure IPsec

current_peer 52.x.x.x port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 0

** I don’t know what is the vendor on the remote end, but there are two ways to configure IPsecoverGRE on Cisco Router:

1.Crypto map Based:

https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/9221-quicktip.html

 2. Tunnel Interface Based:

http://henrydu.com/blog/networks/vpn/ipsec-over-gre-and-ipsec-vti-368.html

** Phase-2 for the tunnel is UP so I think the remote end is configured to crypto map based IPsecoverGRE with matching ACL for traffic permit 47 10.7.64.32/255.255.255.252 10.100.0.0/255.255.255.240.

** You see the tunnel up because remote end is the “initiator” and tunnel came up with IPsec SA proposed by remote end, as IPsecSA on your end is a superset of IPsecSA (0.0.0.0/0.0.0.0/0/0) on remote end.

Simple recommendation would be to match the tunnel config method (VTI or crypto map based) on both end of the tunnel.

./Adesh

[kamrannaseem1]

Hello Adesh,

Thank you very much for your help.

When I told the remote end to change the tunnel type at their end from GRE to IPsec it started working, though we cannot ping the tunnel IP addresses on either end of the tunnel but we can ping the LAN IPs on both end and could also see the packet encaps and decaps. 

The remote end is using strong Swan.

But we are still seeing two tunnels why is that ?

Cisco#sh crypto session ivrf SimulationSystems-106816 detail
Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect

Interface: Tunnel1751
Profile: SimulationSystems-106816-profile-A
Uptime: 07:37:02
Session status: UP-ACTIVE
Peer: 52.x.x.x port 4500 fvrf: (none) ivrf: SimulationSystems-106816
Phase1_id: 52.x.x.x
Desc: (none)
Session ID: 655310
IKEv2 SA: local 95.x.x.x/4500 remote 52.x.x.x/4500 Active
Capabilities:N connid:6 lifetime:00:22:58
IPSEC FLOW: permit ip 10.7.64.32/255.255.255.252 10.100.0.0/255.255.255.240
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 10 drop 0 life (KB/Sec) 4260531/23 hours, 42 mins
Outbound: #pkts enc'ed 10 drop 0 life (KB/Sec) 4260531/23 hours, 42 mins

Interface: Tunnel1751
Session status: DOWN
Peer: 52.x.x.x port 500 fvrf: (none) ivrf: SimulationSystems-106816
Desc: (none)
Phase1_id: (none)
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 0/0

[agairola]

Hi,

Its happening cause they are using crypto map based tunnel using below ACL:

permit ip 10.7.64.32/255.255.255.252 10.100.0.0/255.255.255.240

Which does not contain your tunnel interface IP, tunnel ips communication is not using tunnel.

Simply ask them to switch to using "tunnel protection", as you are doing it on your end.

./Adesh

[Bogdan Nita]

It seems to me you have a VTI tunnel there.

Can you ping the other end of the tunnel 169.250.20.22 ?

It seems you are trying to ping 10.100.0.0/28.

Not encapsulating packets from your side usually indicate a routing or NAT problem on your device.

Did you try to specify the next hop IP (169.250.20.22) for the 10.100.0.0/28 route ?

[kamrannaseem1]

Hello,
I cannot ping the other end 169.250.20.22
I have static route. Is that ok ?
ip route vrf SimulationSystems-106816 10.100.0.0 255.255.255.240 Tunnel1751

[Bogdan Nita]

Not being able to ping the other side of the VTI tunnel, means the tunnel has some issues or the other end is not responding to pings, but you should be able to see encrypted packets.

Do you see encrypted packets when pinging 169.250.20.22 ?

Can you post the output from sh crypto ipsec sa and sh interface Tunnel1751 ?

[kamrannaseem1]

Hello,
I do not see any encrypted packets wen I ping 169.250.20.22.
I have already posted the sh crypto ipsec and interface Tunnel1751 info above.

[Bogdan Nita]

and another thing...

Do you specify the vfr when pinging ?

[kamrannaseem1]

Yes I do specify vrf when I ping.