Re: CSR1000v routing issue - ping failing across router from server

MDBee · ‎12-11-2024

Hi, I am having an issue pinging from one server to another server through a IPSEC tunnel on a Cisco CSR1000v router.

ServerA (10.1.1.2) -> Router (10.1.0.1) / IPSEC Tunnels -> ServerB (10.1.2.2)
-> ServerC (10.1.2.5)

ServerA can ping the router successfully. ServerB can ping the router successfully. The router can ping both servers successfully and Server B can ping Server A successfully so the only issue is from Server A to Server B. There are no firewalls between servers and no access-lists blocking the traffic. The weird thing is Server A can ping Server C which is on the same subnet as Server B successfully. I have checked the windows firewalls on Servers B and C and ICMP is allowed on both so no difference there. Servers B and C are both in AWS and again I noticed no security group differences between them that would cause one to return pings to Server A and not the other one.

I think the issue must be on the Router but I can't see a what it could be. When on the router if I run a show ip route to Server B and C it shows the same route below which send traffic accross the IPSEC Tunnel.

show ip route 10.1.2.1
Routing entry for 10.1.2.0/24
Known via "bgp ", distance 20, metric 100
Tag , type external
Last update from 169.254.11.5 2d00h ago
Routing Descriptor Blocks:
* 169.254.11.5, from 169.254.11.5, 2d00h ago
opaque_ptr 0x7FABEB5B5148
Route metric is 100, traffic share count is 1
AS Hops 1
Route tag
MPLS label: none

Any suggestions would be much appreciated. I can provide more details.

Thanks,
M

Flavio Miranda · ‎12-11-2024

@MDBee

Can we see the show running-config?

This is a site to site VPN? or client-to-site and server B and C is client?

MDBee · ‎12-12-2024

Hi,

It is a site to site vpn. I have pasted the running-config below.

!
version 17.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname
!
boot-start-marker
boot system bootflash:packages.conf
boot-end-marker
!
!
vrf definition GS
rd 100:100
!
address-family ipv4
exit-address-family
!
logging buffered 1000000
logging persistent size 1000000 filesize 8192
!
aaa new-model
!
!
aaa group server radius SSLVPN_RADIUS
server name ISE01
server name ISE02
!
aaa authentication login default local
aaa authentication login sslvpn group SSLVPN_RADIUS local
aaa authentication login sslvpn_radius group SSLVPN_RADIUS local
aaa authorization exec default local
aaa authorization network sslvpn group SSLVPN_RADIUS local
!
!
!
!
!
!
aaa session-id common
!
!
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!

redundancy
!
!

!
!
crypto isakmp policy 200
encryption aes
authentication pre-share
group 2
lifetime 28800
!
crypto isakmp policy 201
encryption aes
authentication pre-share
group 2
lifetime 28800
!
crypto isakmp policy 500
encryption aes 256
authentication pre-share
group 2
lifetime 28800
crypto isakmp keepalive 10 10
crypto isakmp profile isakmp-vpn-0
keyring keyring-vpn-0
match identity address "IP removed"
local-address 10.1.0.1
crypto isakmp profile isakmp-vpn-1
keyring keyring-vpn-1
match identity address "IP removed"
local-address 10.1.0.1
!
crypto ipsec security-association replay window-size 128
!
crypto ipsec transform-set ipsec-prop-vpn-0 esp-aes esp-sha-hmac
mode tunnel
crypto ipsec transform-set ipsec-prop-vpn-1 esp-aes esp-sha-hmac
mode tunnel
crypto ipsec df-bit clear
no crypto ipsec nat-transparency udp-encapsulation
!
!
crypto ipsec profile ipsec-vpn-0
set transform-set ipsec-prop-vpn-0
set pfs group2
!
crypto ipsec profile ipsec-vpn-1
set transform-set ipsec-prop-vpn-1
set pfs group2
!
!
!
!
!
interface Tunnel1
ip address 169.254.11.2 255.255.255.252
ip nat inside
ip tcp adjust-mss 1379
tunnel source 10.1.0.1
tunnel mode ipsec ipv4
tunnel destination "ip removed"
tunnel protection ipsec profile ipsec-vpn-1
ip virtual-reassembly
!
interface Tunnel2
ip address 169.254.11.6 255.255.255.252
ip nat inside
ip tcp adjust-mss 1379
tunnel source 10.1.0.1
tunnel mode ipsec ipv4
tunnel destination "ip removed"
tunnel protection ipsec profile ipsec-vpn-0
ip virtual-reassembly
!
interface GigabitEthernet1
ip address dhcp
ip nat outside
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet2
ip address 10.1.0.2 255.255.255.240
ip nat inside
ip ospf 1 area 0
negotiation auto
no mop enabled
no mop sysid
!
interface GigabitEthernet3
ip address 10.1.0.3 255.255.255.240
ip nat inside
negotiation auto
no mop enabled
no mop sysid
!
router ospf 1
passive-interface default
no passive-interface GigabitEthernet2
!
router bgp
bgp log-neighbor-changes
neighbor 10.1.0.70 remote-as
neighbor 10.1.0.70 description
neighbor 169.254.11.1 remote-as
neighbor 169.254.11.1 timers 10 30 30
neighbor 169.254.11.5 remote-as
neighbor 169.254.11.5 timers 10 30 30
!
address-family ipv4
network 0.0.0.0
network 172.16.0.0 mask 255.255.224.0
network 172.16.0.0 mask 255.248.0.0
network 172.18.0.0 mask 255.255.252.0
network 172.20.0.0 mask 255.255.252.0
network 172.20.11.0 mask 255.255.255.0
network 172.20.12.0 mask 255.255.252.0
network 172.20.24.0 mask 255.255.252.0
neighbor 10.1.0.70 activate
neighbor 10.1.0.70 soft-reconfiguration inbound
neighbor 169.254.11.1 activate
neighbor 169.254.11.1 default-originate
neighbor 169.254.11.1 soft-reconfiguration inbound
neighbor 169.254.11.1 route-map DEFAULT_OUT_PL out
neighbor 169.254.11.5 activate
neighbor 169.254.11.5 default-originate
neighbor 169.254.11.5 soft-reconfiguration inbound
neighbor 169.254.11.5 route-map DEFAULT_OUT_PL out
!
virtual-service csr_mgmt
ip shared host-interface GigabitEthernet1
!
iox
ip forward-protocol nd
ip tcp window-size 8192
no ip http server
ip http authentication local
no ip http secure-server
ip http tls-version TLSv1.2
!
ip nat inside source static tcp 10.1.2.157 80 10.1.0.11 80 extendable
ip nat inside source static tcp 10.1.2.157 443 10.1.0.11 443 extendable
ip nat inside source static tcp 10.1.2.99 514 10.1.0.11 514 extendable
ip nat inside source list NAT interface GigabitEthernet1 overload
ip nat inside source list GS_NAT_ACL interface GigabitEthernet1 vrf GS overload
ip ftp source-interface GigabitEthernet1
ip tftp source-interface GigabitEthernet1
ip route 0.0.0.0 0.0.0.0 GigabitEthernet1 10.1.7.1
ip route vrf GS 0.0.0.0 0.0.0.0 GigabitEthernet1 10.1.7.1 global
ip ssh source-interface GigabitEthernet1C
ip ssh server algorithm mac hmac-sha2-512 hmac-sha2-256
ip ssh server algorithm encryption aes256-ctr aes256-gcm
ip ssh server algorithm kex ecdh-sha2-nistp384 ecdh-sha2-nistp521
ip ssh server algorithm hostkey rsa-sha2-512 rsa-sha2-256
ip ssh server algorithm publickey ecdsa-sha2-nistp521 ecdsa-sha2-nistp384
ip scp server enable
!
ip access-list standard GS_NAT_ACL
10 permit 192.168.35.0 0.0.0.255
ip access-list standard sslvpn-tunnel
10 permit 10.1.0.0 0.0.255.255
20 permit 10.191.0.0 0.0.255.255
30 permit 172.16.0.0 0.7.255.255
40 permit 129.147.12.0 0.0.1.255
50 permit 129.147.14.0 0.0.0.255
!
ip access-list extended CAP1-FILTER-LIST
10 permit icmp any any
ip access-list extended CAP2-FILTER-LIST
10 permit icmp any any
ip access-list extended MGMT_ACL
10 permit ip 10.0.0.0 0.255.255.255 any
20 permit ip 192.168.0.0 0.0.255.255 any
30 permit ip 172.16.0.0 0.15.255.255 any
40 deny ip any any log
ip access-list extended NAT
10 permit ip 10.1.2.0 0.0.0.255 any
!
!
ip prefix-list DEFAULT_OUT_PL seq 5 permit 0.0.0.0/0
!
!
logging trap debugging
logging source-interface GigabitEthernet1
logging host 10.1.0.71
logging host 10.1.2.99
!
!
route-map DEFAULT_OUT permit 10
match ip address prefix-list DEFAULT_OUT_PL
!
!

MDBee · ‎12-19-2024

@Flavio Miranda Did you think of anything after looking at the config?

Flavio Miranda · ‎12-19-2024

when it comes to VPN S2S, we need to see both sides. But, what do you see with the command

show crypto isakmp sa and show crypto ipsec sa

Do you have Phase1 at least?

MDBee · ‎12-19-2024

Hi @Flavio Miranda., I have pasted the output of those commands below.

IPv4 Crypto ISAKMP SA
dst src state conn-id status
10.1.0.1 18.*.*.* QM_IDLE 47089 ACTIVE isakmp-vpn-0
10.1.0.1 18.*.*.* QM_IDLE 47085 ACTIVE isakmp-vpn-1

#show crypto ipsec sa

interface: Tunnel1
Crypto map tag: Tunnel1-head-0, local addr 10.1.0.1

protected vrf: (none)
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
current_peer 18.*.*.* port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 1912011455, #pkts encrypt: 1912011455, #pkts digest: 1912011455
#pkts decaps: 968026482, #pkts decrypt: 968026482, #pkts verify: 968026482
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 31941

local crypto endpt.: 10.1.0.1, remote crypto endpt.: 18.*.*.*
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet1
current outbound spi: 0xCAB8C46B(3401106539)
PFS (Y/N): Y, DH group: group2

inbound esp sas:
spi: 0x442B96E(71481710)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 33249, flow_id: CSR:31249, sibling_flags FFFFFFFF80004048, crypto map: Tunnel1-head-0
sa timing: remaining key lifetime (k/sec): (4607947/725)
IV size: 16 bytes
replay detection support: Y replay window size: 128
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xCAB8C46B(3401106539)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 33250, flow_id: CSR:31250, sibling_flags FFFFFFFF80004048, crypto map: Tunnel1-head-0
sa timing: remaining key lifetime (k/sec): (4607971/725)
IV size: 16 bytes
replay detection support: Y replay window size: 128
Status: ACTIVE(ACTIVE)

outbound ah sas:
outbound pcp sas:

interface: Tunnel2
Crypto map tag: Tunnel2-head-0, local addr 10.1.0.1

protected vrf: (none)
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
current_peer 18.*.*.* port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 301789277, #pkts encrypt: 301789277, #pkts digest: 301789277
#pkts decaps: 380912611, #pkts decrypt: 380912611, #pkts verify: 380912611
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 207

local crypto endpt.: 10.1.0.1, remote crypto endpt.: 18.*.*.*
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet1
current outbound spi: 0xC3F78AB0(3287780016)
PFS (Y/N): Y, DH group: group2

inbound esp sas:
spi: 0x3459E622(878306850)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 33259, flow_id: CSR:31259, sibling_flags FFFFFFFF80004048, crypto map: Tunnel2-head-0
sa timing: remaining key lifetime (k/sec): (4495901/2213)
IV size: 16 bytes
replay detection support: Y replay window size: 128
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xC3F78AB0(3287780016)
transform: esp-aes esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 33260, flow_id: CSR:31260, sibling_flags FFFFFFFF80004048, crypto map: Tunnel2-head-0
sa timing: remaining key lifetime (k/sec): (4538905/2213)
IV size: 16 bytes
replay detection support: Y replay window size: 128
Status: ACTIVE(ACTIVE)

outbound ah sas:
outbound pcp sas:

Flavio Miranda · ‎12-19-2024

Sorry to make you waste your time. You actually dont have a VPN problem. We can see this here:

#pkts encaps: 1912011455, #pkts encrypt: 1912011455, #pkts digest: 1912011455
#pkts decaps: 968026482, #pkts decrypt: 968026482, #pkts verify: 968026482

And we can also see this here

ServerA (10.1.1.2) -> Router (10.1.0.1) / IPSEC Tunnels -> ServerB (10.1.2.2)
-> ServerC (10.1.2.5)

As Server A can ping C.

What I can see is that, considering we are looking at the A side, the Server have a layer3 gateway which is not the router. The router does not have interface on 10.1.1.x. This probably applies to the B/C side.

Did yo check those devices?

If you run a traceroute from A to B and from A to C, how this looks like?

MDBee · ‎12-23-2024

Annoyingly both traceroutes appear the same the both show the first hop to the router and then it just timesout.
traceroute 10.1.2.2
traceroute to 10.1.2.2 (10.1.2.2), 30 hops max, 60 byte packets
1 10.1.0.1 (10.1.0.1) 0.836 ms 0.773 ms 0.744 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

I'm not sure why a ping works from A to C but traceroute doesn't show the full route. ServerA (10.1.1.2) is a FMC if that gives you anything else to go on.

M02@rt37 · ‎12-11-2024

Hello @MDBee

Provide please these outputs:

debug crypto ipsec
debug crypto isakmp

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

MDBee · ‎12-12-2024

Hi,
The logs are constant so not sure if they are useful or not. There are a few active IPSEC tunnels on this Router. I tried to copy and paste debug logs here but cisco prevented me from uploading the post. I removed all IP information but there must have been something I pasted that the forum still didn't like.
Thanks,
M