Hello,
We are facing weird issue, suddenly working VPN went down, while reviewing, we found that tunnel is up however traffic is not passing. I can see pkts encap:0 and pkts decaps increasing. I have verified ACL/NAT thoroughly but unable to find the root cause.
Additionally, packet tracer also looks good. Urgent help is required. Here are some Environment details and some outputs
Environment:
OnPrem
Cisco ASA 5545
ASA Ver: 9.2
ASDM Ver: 7.2
ASA Interesting Network: 10.20.31.0/24
Azure Side
Routed :
VNET Interesting Traffic: 10.20.80.0/20
#show crypto ipsec sa peer AZ.AZ.AZ.AZ
peer address: AZ.AZ.AZ.AZ
Crypto map tag: Outside-W_map, seq num: 56, local addr: ASA-ASA-ASA-ASA
access-list Outside-W_cryptomap_51 extended permit ip 10.20.31.0 255.255.255.0 10.20.80.0 255.255.240.0
local ident (addr/mask/prot/port): (10.20.31.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.20.80.0/255.255.240.0/0/0)
current_peer: AZ.AZ.AZ.AZ
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 9722, #pkts decrypt: 9722, #pkts verify: 9722
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0
local crypto endpt.: ASA-ASA-ASA-ASA/500, remote crypto endpt.: AZ.AZ.AZ.AZ/500
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 2BB3FCD4
current inbound spi : F48DFC9C
inbound esp sas:
spi: 0xF48DFC9C (4102945948)
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 142692352, crypto-map: Outside-W_map
sa timing: remaining key lifetime (kB/sec): (4331501/2054)
IV size: 16 bytes
replay detection support: N
outbound esp sas:
spi: 0x2BB3FCD4 (733215956)
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 142692352, crypto-map: Outside-W_map
sa timing: remaining key lifetime (kB/sec): (4285440/2054)
IV size: 16 bytes
replay detection support: N
show vpn-sessiondb detail l2l filter ipaddress AZ.AZ.AZ.AZ
Session Type: LAN-to-LAN Detailed
Connection : AZ.AZ.AZ.AZ
Index : 34837 IP Addr : AZ.AZ.AZ.AZ
Protocol : IKEv2 IPsec
Encryption : IKEv2: (1)AES256 IPsec: (2)AES256
Hashing : IKEv2: (1)SHA1 IPsec: (2)SHA1
Bytes Tx : 4413844 Bytes Rx : 11750792
Login Time : 21:18:53 EDT Tue May 2 2023
Duration : 11h:27m:01s
IKEv2 Tunnels: 1
IPsec Tunnels: 2
IKEv2:
Tunnel ID : 34837.1
UDP Src Port : 500 UDP Dst Port : 500
Rem Auth Mode: preSharedKeys
Loc Auth Mode: preSharedKeys
Encryption : AES256 Hashing : SHA1
Rekey Int (T): 28800 Seconds Rekey Left(T): 14948 Seconds
PRF : SHA1 D/H Group : 2
Filter Name :
IPsec:
Tunnel ID : 34837.3
Local Addr : 10.20.31.0/255.255.255.0/0/0
Remote Addr : 10.20.80.0/255.255.240.0/0/0
Encryption : AES256 Hashing : SHA1
Encapsulation: Tunnel
Rekey Int (T): 3600 Seconds Rekey Left(T): 1406 Seconds
Rekey Int (D): 4608000 K-Bytes Rekey Left(D): 4607974 K-Bytes
Idle Time Out: 30 Minutes Idle TO Left : 29 Minutes
Bytes Tx : 0 Bytes Rx : 500492
Pkts Tx : 0 Pkts Rx : 9878
IPsec:
Tunnel ID : 34837.4
Local Addr : 10.20.31.0/255.255.255.0/0/0
Remote Addr : 10.35.0.0/255.255.0.0/0/0
Encryption : AES256 Hashing : SHA1
Encapsulation: Tunnel
Rekey Int (T): 3600 Seconds Rekey Left(T): 3301 Seconds
Rekey Int (D): 4608000 K-Bytes Rekey Left(D): 4607965 K-Bytes
Idle Time Out: 30 Minutes Idle TO Left : 30 Minutes
Bytes Tx : 2725786 Bytes Rx : 6715217
Pkts Tx : 49620 Pkts Rx : 48589
packet-tracer input inside-D tcp 10.20.31.130 111 10.20.80.11 80
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list
Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 via 212.X.X.X, Outside-W
Phase: 4
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (Inside-D,Outside-W) source static NETWORK_OBJ_10.20.31.0_24 NETWORK_OBJ_10.20.31.0_24 destination static DM_INLINE_NETWORK_32 DM_INLINE_NETWORK_32 no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Outside-W
Untranslate 10.20.80.11/80 to 10.20.80.11/80
Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group Inside-access-IN in interface Inside-D
access-list Inside-access-IN extended permit ip any any
Additional Information:
Phase: 6
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
match any
policy-map global_policy
class class-default
set connection decrement-ttl
service-policy global_policy global
Additional Information:
Phase: 7
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (Inside-D,Outside-W) source static NETWORK_OBJ_10.20.31.0_24 NETWORK_OBJ_10.20.31.0_24 destination static DM_INLINE_NETWORK_32 DM_INLINE_NETWORK_32 no-proxy-arp route-lookup
Additional Information:
Static translate 10.20.31.130/111 to 10.20.31.130/111
Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 10
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Phase: 11
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Phase: 12
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (Inside-D,Outside-W) source static NETWORK_OBJ_10.20.31.0_24 NETWORK_OBJ_10.20.31.0_24 destination static DM_INLINE_NETWORK_32 DM_INLINE_NETWORK_32 no-proxy-arp route-lookup
Additional Information:
Phase: 13
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Phase: 14
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Phase: 15
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Phase: 16
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 17
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Phase: 18
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 1033661206, packet dispatched to next module
Result:
input-interface: Inside-D
input-status: up
input-line-status: up
output-interface: Outside-W
output-status: up
output-line-status: up
Action: allow
