Cisco 2811 site-to-site ipsec fragmentation problem

Mcicoolll · ‎02-22-2017

Hello cisco community, i have a packet drop problem in my site-to-site vpns. Here is my topology:

Hosts (say 192.168.1.0/24) ->L3 switch->Cisco 2811 (192.168.1.5) -> INET ----> IPSEC TUNNEL ----> Cisco 2811 (192.168.2.5)-> Mikrotik router -> Hosts in other office (192.168.2.0/24)

When i do simple pings from my host in 1.0/24 to cisco 2.5 i see packet drops from time to time. The channel provided by ISP is 20 mbits but inside the tunnel as soon as the speeds go up to 6-7 mbits packets start to drop.

I am now so tired to solve this problem... where can be the issue?

P.S. If i connect to one of the 2811s via ssh and try to ping other 2811 or any host in its subnet - i cant do this. but pings from any host to any host works fine. - Is this issue connected to my main problem?

Cisco 1 interfaces:

interface FastEthernet0/0
description == WAN ==
ip address x.x.x.x 255.255.255.224
ip access-group 103 in
ip verify unicast reverse-path
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip inspect CCP_LOW out
ip virtual-reassembly
duplex auto
speed auto
no cdp enable
no mop enabled
crypto map SDM_CMAP_1
!
interface FastEthernet0/1
description == LAN ==
ip address 192.168.1.5 255.255.255.0
ip access-group 105 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip tcp adjust-mss 1400
duplex auto
speed auto
no cdp enable
no mop enabled

Cisco 2 interfaces:

interface FastEthernet0/0
description === EXTERNAL ===
ip address y.y.y.y 255.255.255.0
ip access-group 105 in
ip verify unicast reverse-path
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip inspect CCP_LOW out
ip virtual-reassembly
duplex auto
speed auto
no cdp enable
no mop enabled
crypto map SDM_CMAP_1
!
interface FastEthernet0/1
description === INTERNAL ===
ip address 192.168.2.5 255.255.255.0
ip access-group 104 in
ip verify unicast reverse-path
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip tcp adjust-mss 1400
duplex auto
speed auto
no cdp enable
no mop enabled

Rahul Govindan · ‎02-22-2017

Since these are packet-drops, it is unlikely that there is a wrong config on the routers. I would do the following to troubleshoot this further.

1) Create a separate ACL for host on side A to Side B for testing purpose and place it above existing LAN A to Lan B ACL. This way you can see the "show crypto ipsec sa" output between these hosts alone - which will help determine which side has the problem.

2) If you are running 15.x version of the router, you can run pack-captures on the LAN and WAN side for unencrypted and encrypted traffic. This way you can see all sides of the transactions.

3) Eliminate hops if possible. This will help isolate the issue to a single device or the ISP.

Mcicoolll · ‎02-22-2017

Router version is 12.x so i will stick to option 1. Before that here is the output of current "sh cry ipsec sa" on Router-2 (#send errors 2, #recv errors 5534 scares me):

protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
   current_peer x.x.x.x port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 774226, #pkts encrypt: 774226, #pkts digest: 774226
    #pkts decaps: 875804, #pkts decrypt: 875804, #pkts verify: 875804
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 2, #recv errors 5534

     local crypto endpt.: y.y.y.y, remote crypto endpt.: x.x.x.x
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0x3932C608(959628808)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0xE6E643DF(3873850335)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2525, flow_id: NETGX:525, sibling_flags 80000046, crypto map: SDM_CMAP_1
        sa timing: remaining key lifetime (k/sec): (3948449/586)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x3932C608(959628808)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2526, flow_id: NETGX:526, sibling_flags 80000046, crypto map: SDM_CMAP_1
        sa timing: remaining key lifetime (k/sec): (4293442/586)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:

Rahul Govindan · ‎02-22-2017

Try doing a "show crypto ipsec sa detail" to see details of what type of send and rcv errors are seen.

Mcicoolll · ‎02-22-2017

I am not cool enough to realise what it means... Thank you very much for help..

#pkts encaps: 1438905, #pkts encrypt: 1438905, #pkts digest: 1438905
    #pkts decaps: 1567831, #pkts decrypt: 1567831, #pkts verify: 1567831
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #pkts no sa (send) 2, #pkts invalid sa (rcv) 1
    #pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
    #pkts invalid prot (recv) 0, #pkts verify failed: 0
    #pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
    #pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
    ##pkts replay failed (rcv): 0
    #pkts internal err (send): 0, #pkts internal err (recv) 10817

Rahul Govindan · ‎02-22-2017

The internal recv error usually comes up when you have some QoS or service-policy in place on your WAN interface. I did not see this in your case. Another problem could be the crypto engine itself which might be dropping packets on high load. I would see if this counter increases when you see the issue happening. I can already see that this has increased between your 2 outputs.

You are also running a 12.x version, so you may be hitting issues with that too. See if you can upgrade to the latest 12.4 release or 15.1 release for the 2811 router.

Mcicoolll · ‎02-24-2017

Hm, my version is 12.4(24)T8. Basicle drops start when the flow reaches 6/7 mbits/s

So you say that actually there is nothing i can do with that?

Maybe upgrade to 15.x and increase of RAM to 512? Or this is not connected ot my problem..?

Mcicoolll · ‎02-26-2017

I also forgot to mention the topology on the side-B (right one)..

There is a router Mikrotik which is def gate for hosts and there are static routes on it to redirect VPN-related subnets to 2811.. So basicly packets go inside vpn through Mikrotik, but on the way back Mikrotik is not involved so the route is shorter... Does it matter?