This blog is in continuation with my If incase you are a new user then I would highly encourage you to go through first 2 blogs as well, to have a better understanding.
I also want to thank you for your incredible responses on other blogs.
In this part 2 will be discussing the following problem scenarios----
Problem Scenario 1:
Routing Issues
User complains there is no traffic received through the IPSec tunnel. On further checking you find that IKE and IPSec SAs exist, but no end-end traffic; spoke shows its encrypting traffic however no decrpyt.
Check for IPSec SA on Hub Site (look for inbound and outbound SPIs, encr/decr counts)
HUB# sh crypto session remote 40.10.1.1 detail
Crypto session current status
Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, X - IKE Extended Authentication
Interface: GigabitEthernet0/1
Profile: SPOKE10-PROF
Uptime: 00:01:49
Session status: UP-ACTIVE
Peer: 40.10.1.1 port 500 fvrf: (none) ivrf: (none)
Phase1_id: 40.10.1.1
Desc: (none)
IKE SA: local 30.3.1.1/500 remote 40.10.1.1/500 Active
Capabilities:D connid:1029 lifetime:01:58:10
IPSEC FLOW: permit ip 3.1.1.0/255.255.255.0 4.1.1.0/255.255.255.0
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 9949 drop 60 life (KB/Sec) 4483560/1690
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4485046/1690
HUB# sh crypto ipsec sa peer 40.10.1.1
interface: GigabitEthernet0/1
Crypto map tag: CMAP, local addr 30.3.1.1
protected vrf: (none)
local ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
current_peer 40.10.1.1 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 9949, #pkts decrypt: 9949, #pkts verify: 9949
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 60
local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
current outbound spi: 0xF6278D63(4129787235)
inbound esp sas:
spi: 0x16C58DD4(382045652)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 27, flow_id: SW:27, crypto map: CMAP
sa timing: remaining key lifetime (k/sec): (4483560/1659)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0xF6278D63(4129787235)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 28, flow_id: SW:28, crypto map: CMAP
sa timing: remaining key lifetime (k/sec): (4485046/1657)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
Check the Routes (for the Spoke protected networks)
HUB# sh ip route 4.1.1.1
% Network not in table
HUB#
HUB# sh ip cef 4.1.1.1
0.0.0.0/0, version 80, epoch 0, attached, default route handler
0 packets, 0 bytes
via 0.0.0.0, 0 dependencies
valid no route adjacency
HUB#
Check the Crypto Map for Reverse-Route Injection. This is needed for the Hub to inject a route for the Spoke protected subnets into its local routing table. The route is created when the IPSec SA is established. Since 12.4T, for this route to be created (based on the Crypto ACL) before the IPSec SA is established (so that the router can initiate the tunnel), we need the “reverse-route static” configuration.
In the VRF-Aware IPSec scenario, it is better to use the “reverse-route remote-peer <next-hop-gateway>” configuration under the crypto map.
Old Crytpo Map was----
crypto map CMAP 10 ipsec-isakmp
set peer 40.10.1.1
set transform-set TS
set isakmp-profile SPOKE10-PROF
match address SPOKE10-ACL
Lets add reverse route---
crypto map CMAP 10 ipsec-isakmp
set peer 40.10.1.1
set transform-set TS
set isakmp-profile SPOKE10-PROF
match address SPOKE10-ACL
reverse-route <static>
IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer 40.10.1.1
IPSEC(rte_mgr): VPN Route Event create SA based on crypto ACL in real time for 40.10.1.1
IPSEC(rte_mgr): VPN Route Refcount 1 GigabitEthernet0/1
IPSEC(rte_mgr): VPN Route Added 4.1.1.0 255.255.255.0 via 0.0.0.0 in IP DEFAULT TABLE with tag 0 distance 1
HUB# sh ip route 4.1.1.1
Routing entry for 4.1.1.0/24
Known via "static", distance 1, metric 0
Redistributing via ospf 1
Advertised by ospf 1 subnets
Routing Descriptor Blocks:
* 40.10.1.1
Route metric is 0, traffic share count is 1
HUB# sh ip cef 4.1.1.1
4.1.1.0/24, version 83, epoch 0, cached adjacency 30.3.1.2
0 packets, 0 bytes
via 40.10.1.1, 0 dependencies, recursive
next hop 30.3.1.2, GigabitEthernet0/1 via 40.0.0.0/8
valid cached adjacency
Complete Crypto Map
HUB# show crypto map
Crypto Map "CMAP" 10 ipsec-isakmp
Peer = 40.10.1.1
ISAKMP Profile: SPOKE10-PROF
Extended IP access list SPOKE10-ACL
access-list SPOKE10-ACL permit ip 3.1.1.0 0.0.0.255 4.1.1.0 0.0.0.255
Current peer: 40.10.1.1
Security association lifetime: 4608000 kilobytes/1800 seconds
PFS (Y/N): N
Transform sets={
TS,
}
Reverse Route Injection Enabled
Interfaces using crypto map CMAP:
GigabitEthernet0/1
Problem Scenario 2:
DPD
This is a scenario where HUB keeps sending encrypted traffic, but it is not receiving any encrypted traffic from Spoke. IKE and IPSec SAs are up.
Please perform the following steps------
Check if the Spoke is reachable (ping tunnel endpoint address)
HUB# ping 40.10.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 40.10.10.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
No replies.
Check if Dead peer Detection is turned on
HUB# sh cry isa sa detail
Codes: C - IKE configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal
X - IKE Extended Authentication
psk - Preshared key, rsig - RSA signature
renc - RSA encryption
IPv4 Crypto ISAKMP SA
C-id Local Remote I-VRF Status Encr Hash Auth DH Lifetime Cap.
1035 30.3.1.1 40.10.1.1 ACTIVE 3des sha psk 2 01:59:45
Engine-id:Conn-id = SW:35
So Under Cap, its not listing anything, hence it is disabled.
Check the IPSec SA
HUB#sh cry ips sa
interface: GigabitEthernet0/1
Crypto map tag: CMAP, local addr 30.3.1.1
protected vrf: (none)
local ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
current_peer 40.10.1.1 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 100, #pkts encrypt: 100, #pkts digest: 100
#pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
current outbound spi: 0x8BDBBA86(2346433158)
inbound esp sas:
spi: 0x67C89AAD(1741200045)
We notice very few decaps, lets go ahead and configure DPD.
Configure DPD
crypto isakmp keepalive 60 10
crypto isakmp keepalive 60 periodic
If DPD had been configured earlier, then you would have seen following-----
HUB# sh cry isakmp peer de
Peer: 40.10.1.1 Port: 500 Local: 30.3.1.1
Phase1 id: 40.10.1.1
flags:
NAS Port: 0 (Normal) DPD information, struct 0x6727E0E8:
Last_received: 237, dpd threshold (elapsed) 0
my_last_seq_num: 0x5B72ECCC, peers_last_seq_num: 0x0
sent_and_waiting: TRUE
IKE SAs: 1 IPSec SA bundles: 1
last_locker: 0x62FE32FC, last_last_locker: 0x0
last_unlocker: 0x0, last_last_unlocker: 0x0
HUB# sh cry isa sa detail
Codes: C - IKE configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal
X - IKE Extended Authentication
psk - Preshared key, rsig - RSA signature
renc - RSA encryption
IPv4 Crypto ISAKMP SA
C-id Local Remote I-VRF Status Encr Hash Auth DH Lifetime Cap.
1037 30.3.1.1 40.10.1.1 ACTIVE 3des sha psk 2 01:59:43 D
Engine-id:Conn-id = SW:37
PS: It shows D under Cap now.
ISAKMP:(1036):DPD incrementing error counter (4/5)
ISAKMP: set new node 1992211651 to QM_IDLE
ISAKMP:(1036):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 1718567840, message ID = 1992211651
ISAKMP:(1036): seq. no 0x5B72ECCD
ISAKMP:(1036): sending packet to 40.10.1.1 my_port 500 peer_port 500 (R) QM_IDLE
ISAKMP:(1036):Sending an IKE IPv4 Packet.
ISAKMP:(1036):purging node 1992211651
ISAKMP:(1036):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
ISAKMP:(1036):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
ISAKMP:(1036):DPD incrementing error counter (5/5)
ISAKMP:(1036):peer 40.10.1.1 not responding!
ISAKMP:(1036):peer does not do paranoid keepalives.
ISAKMP:(1036):deleting SA reason "P1 errcounter exceeded (PEERS_ALIVE_TIMER)" state (R) QM_IDLE (peer 40.10.1.1)
It is always better to use DPD instead of Periodic Keepalives. DPD works well in conjunction with IPSec HA – geographically distributed peers (multiple ‘set peer’ under crypto map), or HSRP adjacent peers (peer to VIP address).
Problem Scenario 3:
Anti-Replay Issues
Users complain that application is losing intermittent traffic, or that Voice quality through tunnel is bad. Check if the IPSec SA is showing anti-replay drops
HUB# sh cry ips sa peer 40.10.1.1 detail
interface: GigabitEthernet0/1
Crypto map tag: CMAP, local addr 30.3.1.1
protected vrf: (none)
local ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
current_peer 40.10.1.1 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 2900, #pkts encrypt: 2900, #pkts digest: 2900
#pkts decaps: 1909, #pkts decrypt: 1909, #pkts verify: 1909
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#pkts no sa (send) 0, #pkts invalid sa (rcv) 0
#pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
#pkts invalid prot (recv) 0, #pkts verify failed: 0
#pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
#pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
##pkts replay failed (rcv): 1000
#pkts internal err (send): 0, #pkts internal err (recv) 0
local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1
path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
current outbound spi: 0xC37422AA(3279168170)
inbound esp sas:
spi: 0x135E76B1(324957873)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 41, flow_id: SW:41, crypto map: CMAP
sa timing: remaining key lifetime (k/sec): (4419198/860)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
So we can see there is a good number of packets replay failed in above show commands.
By Default IPSec Anti-Replay window size is 64. Hence, Packets received outside the window will be dropped. Normally Re-ordering of packets could happen due to QoS on the encrypting router (Spoke) or in the Transit Network.
In current Cisco IOS versions, the Anti-Replay window can be increased up to 1024, or diabled altogether
crypto ipsec security-association window-size < Size>
crypto ipsec security-association replay disable
It is not recommended to disable anti-replay. Hence first try to fix the QoS issue in the network or encrypting router; give better QoS to Voice traffic, or use crypto LLQ; then try to increase the anti-replay window size by above mentioned command.
That sums up my problem scenarios blog. Hope you liked these scenarios and you will be able to implement these best practices in your infrastructure to have a seamless network.
Looking forward to your comments and inputs and also please let me know on what topic you want to see future blogs.
Appreciate your time and contribution on CSC
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: