That helps a lot. It's very

bhuwanchandra6315 · ‎03-20-2014

Hi Experts,

I have found one strange problem with IPSec VPN, the scenario is like this, our corporate office is connected to its 25 remote office with IPSec VPN, at corporate site, cisco 2811 router is installed and same type of router is installed at each remote site and IPSec VPN is configured between remote office and corporate office and further each remote site router has two other VPN configured which are working properly. Now the problem is, 4 out of 25 remote offices are not getting up with corporate office, I mean the VPN is not getting up for these location. I sit at corporate office and have tried my level best to up these VPN but the problem not getting resolved.

Now the strange problem is that the VPN gets up by itself, after sometime like in 10days or 20days, for sometime and gets down by itself later.

Anyone who can give some insights where the problem could be and how could i troubleshoot the problem?

Thanks in advance for your valuable response

Mike Williams · ‎03-20-2014

Do you have any logs or debugs you could share? What about relevant parts of your config? It's very hard for us to troubleshoot without any information to base our suggestions on.

Regards,

Mike

bhuwanchandra6315 · ‎03-20-2014

Hi Mike,

Here is the relavent configuration at my corporate router, however there are 25 tunnels at my corporate router, only some tunnels are mentiond here:

crypto isakmp policy 10
hash md5
authentication pre-share
group 2

crypto isakmp key My_key address remote_ip1
crypto isakmp key My_key address remote_ip2
crypto isakmp key My_key address remote_ip3
!
!
crypto ipsec transform-set My_transform esp-des esp-md5-hmac
!
crypto map My_map 101 ipsec-isakmp
set peer remote_ip1
set transform-set My_transform
match address 101
crypto map My_map 102 ipsec-isakmp
set peer remote_ip2
set transform-set My_transform
match address 102
crypto map My_map 103 ipsec-isakmp
set peer remote_ip3
set transform-set My_transform
match address 103

!
interface Loopback1
ip address 172.21.128.1 255.255.255.255
!
interface FastEthernet0/0
description Towards Internet for VPN
ip address 10.100.103.2 255.255.255.248
ip accounting output-packets
duplex auto
speed auto
crypto map My_map
!

!
ip forward-protocol nd
ip route remote_ip1 255.255.255.255 10.100.103.1
ip route remote_ip2 255.255.255.255 10.100.103.1
ip route remote_ip3 255.255.255.255 10.100.103.1
!

access-list 101 permit ip 172.21.128.0 0.0.3.255 172.20.0.0 0.0.31.255
access-list 102 permit ip 172.21.128.0 0.0.3.255 172.20.128.0 0.0.31.255
access-list 103 permit ip 172.21.128.0 0.0.3.255 172.21.158.0 0.0.1.255

Here is the relavent configuration at my remote office router:

crypto isakmp policy 10
hash md5
authentication pre-share
group 2
crypto isakmp key My_key address remote_ip1
crypto isakmp key My_key address remote_ip2
crypto isakmp key My_key address remote_ip3
!
!
crypto ipsec transform-set My_transform esp-des esp-md5-hmac
!
crypto map My_map 1 ipsec-isakmp
set peer remote_ip1
set transform-set My_transform
match address 101
crypto map My_map 2 ipsec-isakmp
set peer remote_ip2
set transform-set My_transform
match address 102
crypto map My_map 3 ipsec-isakmp
set peer remote_ip3
set transform-set My_transform
match address 100
!
!
!
!
!
!
interface FastEthernet0/0
description Towards Internet for VPN
ip address 10.100.103.122 255.255.255.248
duplex auto
speed auto
crypto map My_map

interface FastEthernet0/1
description Towards Local LAN
ip address 172.21.158.1 255.255.254.0

ip route remote_ip 255.255.255.255 10.100.103.121
ip route remote_ip 255.255.255.255 10.100.103.121
ip route remote_ip 255.255.255.255 10.100.103.121

!
access-list 100 permit ip 172.21.158.0 0.0.1.255 172.20.0.0 0.0.31.255
access-list 101 permit ip 172.21.158.0 0.0.1.255 172.20.128.0 0.0.31.255
access-list 102 permit ip 172.21.158.0 0.0.1.255 172.21.128.0 0.0.3.255
!

here are some logs from corporate router:

Router1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
remote_ip 10.100.103.2 QM_IDLE 1198 0 ACTIVE
remote_ip 10.100.103.2 QM_IDLE 1196 0 ACTIVE
remote_ip 10.100.103.2 MM_NO_STATE 0 0 ACTIVE (deleted)

Router1#sh crypto session remote remote_ip
Crypto session current status

Interface: FastEthernet0/0
Session status: DOWN-NEGOTIATING
Peer: remote_ip port 500
IKE SA: local 10.100.103.2/500 remote remote_ip/500 Inactive
IPSEC FLOW: permit ip 172.21.128.0/255.255.252.0 172.21.158.0/255.255.254.0
Active SAs: 0, origin: crypto map

here are some logs from remote office router:

Router2#sh crypto ipsec sa peer remote_ip

interface: FastEthernet0/0
Crypto map tag: My_map , local addr 10.100.103.2

protected vrf: (none)
local ident (addr/mask/prot/port): (172.21.128.0/255.255.252.0/0/0)
remote ident (addr/mask/prot/port): (172.21.158.0/255.255.254.0/0/0)
current_peer remote_ip port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 25, #recv errors 0

local crypto endpt.: 10.100.103.2, remote crypto endpt.: remote_ip
path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
current outbound spi: 0x0(0)

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

router2#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
remote_ip 10.100.103.122 QM_IDLE 1004 0 ACTIVE
remote_ip 10.100.103.122 QM_IDLE 1003 0 ACTIVE
remote_ip 10.100.103.122 MM_NO_STATE 0 0 ACTIVE
remote_ip 10.100.103.122 MM_NO_STATE 0 0 ACTIVE (deleted)

router2#sh crypto ipsec sa peer remote_ip

interface: FastEthernet0/0
Crypto map tag: My_map, local addr 10.100.103.122

protected vrf: (none)
local ident (addr/mask/prot/port): (172.21.158.0/255.255.254.0/0/0)
remote ident (addr/mask/prot/port): (172.21.128.0/255.255.252.0/0/0)
current_peer remote_ip port 500
PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 6154, #recv errors 0

local crypto endpt.: 10.100.103.122, remote crypto endpt.: remote_ip
path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
current outbound spi: 0x0(0)

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

router2#sh crypto session remote remote_ip
Crypto session current status

Interface: FastEthernet0/0
Session status: DOWN-NEGOTIATING
Peer: remote_ip port 500
IKE SA: local 10.100.103.122/500 remote remote_ip/500 Inactive
IKE SA: local 10.100.103.122/500 remote remote_ip/500 Inactive
IPSEC FLOW: permit ip 172.21.158.0/255.255.254.0 172.21.128.0/255.255.252.0
Active SAs: 0, origin: crypto map

Based on above config and logs could anyone identify, where the problem could be?

Thanks

Bhuwan

Mike Williams · ‎03-21-2014

Are you having any internet connectivity issues at the remote sites?

Can you capture some debugs?

debug crypto condition peer ipv4 *remote peer ip*

debug crypto isakmp

debug crypto engine

debug crypto ipsec

clear log

bhuwanchandra6315 · ‎03-22-2014

Hi Mike,

Thanks for your reply...

Below are some logs from corporate router with one of the tunnel which is not getting up::

RTR-FTR-PJB#debug crypto isakmp
Crypto ISAKMP debugging is on
RTR-FTR-PJB#ping 172.26.10.1 source l1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.26.10.1, timeout is 2 seconds:
Packet sent with a source address of 172.21.128.1

*Mar 22 12:19:32.147: ISAKMP: local port 500, remote port 500
*Mar 22 12:19:32.147: ISAKMP: set new node 0 to QM_IDLE
*Mar 22 12:19:32.147: ISAKMP: Find a dup sa in the avl tree during calling isadb_insert sa = 459BC390
*Mar 22 12:19:32.147: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Mar 22 12:19:32.147: ISAKMP:(0):found peer pre-shared key matching remote_ipsec_peer
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Mar 22 12:19:32.147: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Mar 22 12:19:32.147: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Mar 22 12:19:32.147: ISAKMP:(0):Old State = IKE_READY New State = IKE_I_MM1

*Mar 22 12:19:32.147: ISAKMP:(0): beginning Main Mode exchange
*Mar 22 12:19:32.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:19:32.147: ISAKMP:(0):Sending an IKE IPv4 Packet......
Success rate is 0 percent (0/5)
RTR-FTR-PJB#
*Mar 22 12:19:42.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:19:42.147: ISAKMP (0:0): incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
*Mar 22 12:19:42.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:19:42.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:19:42.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:19:52.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:19:52.147: ISAKMP (0:0): incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Mar 22 12:19:52.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:19:52.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:19:52.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:02.143: ISAKMP: set new node 0 to QM_IDLE
*Mar 22 12:20:02.143: ISAKMP:(0):SA is still budding. Attached new ipsec request to it. (local 10.100.103.2, remote remote_ipsec_peer)
*Mar 22 12:20:02.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:02.147: ISAKMP (0:0): incrementing error counter on sa, attempt 3 of 5: retransmit phase 1
*Mar 22 12:20:02.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:20:02.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:20:02.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:03.847: ISAKMP:(0):purging node 1974447943
*Mar 22 12:20:03.847: ISAKMP:(0):purging node -1277953536
*Mar 22 12:20:12.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:12.147: ISAKMP (0:0): incrementing error counter on sa, attempt 4 of 5: retransmit phase 1
*Mar 22 12:20:12.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:20:12.147: ISAKMP:(0): sending packet to remote_ipsec_peer my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:20:12.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:13.847: ISAKMP:(0):purging SA., sa=451DF344, delme=451DF344
*Mar 22 12:20:22.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:22.147: ISAKMP (0:0): incrementing error counter on sa, attempt 5 of 5: retransmit phase 1
*Mar 22 12:20:22.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar 22 12:20:22.147: ISAKMP:(0): sending packet to remote_ipsec_peermy_port 500 peer_port 500 (I) MM_NO_STATE
*Mar 22 12:20:22.147: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar 22 12:20:32.147: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar 22 12:20:32.147: ISAKMP:(0):peer does not do paranoid keepalives.

*Mar 22 12:20:32.147: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer remote_ipsec_peer)
*Mar 22 12:20:32.147: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer remote_ipsec_peer)
*Mar 22 12:20:32.147: ISAKMP:(0):deleting node -1242602279 error FALSE reason "IKE deleted"
*Mar 22 12:20:32.147: ISAKMP:(0):deleting node 275856152 error FALSE reason "IKE deleted"
*Mar 22 12:20:32.147: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
*Mar 22 12:20:32.147: ISAKMP:(0):Old State = IKE_I_MM1 New State = IKE_DEST_SA

*Mar 22 12:21:22.147: ISAKMP:(0):purging node -1242602279
*Mar 22 12:21:22.147: ISAKMP:(0):purging node 275856152
*Mar 22 12:21:32.147: ISAKMP:(0):purging SA., sa=459BC390, delme=459BC390

RTR-FTR-PJB#debug crypto ipsec
Crypto IPSEC debugging is on
RTR-FTR-PJB#ping 172.26.10.1 source l1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.26.10.1, timeout is 2 seconds:
Packet sent with a source address of 172.21.128.1

*Mar 22 12:23:27.411: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 10.100.103.2, remote= remote_ipsec_peer,
local_proxy= 172.21.128.0/255.255.252.0/0/0 (type=4),
remote_proxy= 172.26.10.0/255.255.254.0/0/0 (type=4),
protocol= ESP, transform= esp-des esp-md5-hmac (Tunnel),
lifedur= 3600s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0.....
Success rate is 0 percent (0/5)
RTR-FTR-PJB#
*Mar 22 12:23:57.411: IPSEC(key_engine): request timer fired: count = 1,
(identity) local= 10.100.103.2, remote= remote_ipsec_peer,
local_proxy= 172.21.128.0/255.255.252.0/0/0 (type=4),
remote_proxy= 172.26.10.0/255.255.254.0/0/0 (type=4)
*Mar 22 12:23:57.411: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 10.100.103.2, remote= remote_ipsec_peer,
local_proxy= 172.21.128.0/255.255.252.0/0/0 (type=4),
remote_proxy= 172.26.10.0/255.255.254.0/0/0 (type=4),
protocol= ESP, transform= esp-des esp-md5-hmac (Tunnel),
lifedur= 3600s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0

RTR-FTR-PJB#debug crypto engine

*Mar 22 12:28:59.415: crypto_engine: Generate IKE hash
*Mar 22 12:28:59.415: crypto_engine: Generate IKE hash
*Mar 22 12:28:59.415: crypto_engine: Encrypt IKE packet
*Mar 22 12:28:59.727: crypto_engine: Generate IKE hash
*Mar 22 12:28:59.727: crypto_engine: Encrypt IKE packet
*Mar 22 12:28:59.763: crypto_engine: Decrypt IKE packet
*Mar 22 12:28:59.763: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.099: crypto_engine: Decrypt IKE packet
*Mar 22 12:29:00.099: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.099: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.099: crypto_engine: Encrypt IKE packet
*Mar 22 12:29:00.239: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.239: crypto_engine: Encrypt IKE packet
*Mar 22 12:29:00.271: crypto_engine: Decrypt IKE packet
*Mar 22 12:29:00.271: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.359: crypto_engine: Decrypt IKE packet
*Mar 22 12:29:00.359: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.359: crypto_engine: Generate IKE hash
*Mar 22 12:29:00.363: crypto_engine: Encrypt IKE packet
*Mar 22 12:29:00.403: crypto_engine: Generate IKE hash

Few things i would like mention here are:

1. I am able to ping remote_ipsec_peer from my router.

2. At both routers other tunnels are working fine.

3. NATing is not involved at both sides router, we have static ip at both side and static routes are configured to reach the peer.

Anyone who can provide some insights by looking the above log, where the problem could be?

Mike Williams · ‎03-24-2014

It appears that router is sending requests to the remote router to establish an ISAKMP session, but not receiving a response to those requests. Can you run the same debugs from the remote router?

Have you tried rebooting the routers for good measure? Could be a bug.

Regards,

Mike

bhuwanchandra6315 · ‎03-27-2014

Router has been reloaded lot of time but no luck and i am not able to the check the debug status from remote router.....

The ipsec vpn was working fine till we have replaced the old ADSL modem with new FTTH modem with new static ip. once the modem and static ip got changed the tunnel never came up, however the other two tunnel got up without any problem.

now i want to the answer of few question?

1. Does the new modem have some problem which is making tunnel down like MTU setting, NAT setting?

2. May be the port 500 is blocked somewhere between ISP devices, in that case how we can check the port staus?

Mike Williams · ‎03-28-2014

That helps a lot. It's very likely that if the DSL and IP were the only things that were changed, then it's related to one of those. I assume you have verified your peer addresses are correct on both ends. I have a couple questions:

1. Are the DSL modems in bridging mode, gateway mode, or PPPoE/PPPoA mode?

2. Are the DSL modems doing any NAT?

If they are in bridging mode, they shouldn't be blocking anything. If they are in gateway mode (block of IPs) or PPPoE/A mode, then they may be blocking IPsec on the integrated firewall.

If you are doing PPPoE/A on the modem, and passing private IP addressing through to the router, then you may need to make some tweaks on the hub router so it uses the private IP of the remote router as the remote router ID. Also, if it's in PPPoE/A mode or bridge mode doing PPPoE/A, then you may need to lower the MTU by 8 bytes on the router, "ip tcp adjust-mss 1492".

Regards,

Mike

bhuwanchandra6315 · ‎03-29-2014

Thanks mike for your valuable points!!!

Peer address are correctly configured at both the ends. i would like to answer your question:

1. DSL/FTTH modem is in PPPoE mode. The modem is a FTTH modem in which FIber is terminated at on end and one of its ethernet port is connected to router.

2. Yes NAT option is enable in modems WAN tab and NAT type is NAPT.

3. As you said we are passing private ip to router, means modem's LAN option and router's connected ethernet port is in same subnet as it is quite obvious.

As i already said the other tunnel are working properly at both end, what should i do?. should i lower the mtu size as you said or somthing else....?

Thanks in advance for you support!!!

bhuwanchandra6315 · ‎04-01-2014

issue still not resolved!!!!!

need some help.......

Thanks

Bhuwan

bhuwanchandra6315 · ‎04-20-2014

Guys, still need your help to resolve the issue....

kindly reply!!!!!

syed kazim abbas · ‎04-21-2014

Hi,

its a trick may be work, in my case it solved.

when u make any changing in configuration then do this on both sides.

interface FastEthernet0/0
description Towards Internet for VPN
ip address 10.100.103.2 255.255.255.248
no crypto map My_map

clear crypto isakmp 1001 (connection-id)

crypto map My-map

4 out of 25 VPN tunnel is not getting up.....