cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
18109
Views
20
Helpful
7
Replies

ASA Site to Site tunnel no transmit traffic for some subnets after a while

Hello,

 

we have a really strange site to site tunnel issue on several ASAs.

We are running VPN tunnels between a small site and three bigger ones.

The small office has an ASA 5505, the other three ones are ASA 5510.

One of the tunnels is working for months without problems.

Each tunnels has several class C network on it.

e.g. Site A:

- 192.168.50.0/24 (named A1)

- 192.168.51.0/24 (named A2)

Site B:

- 192.168.60.0/24 (named B1)

- 192.168.61.0/24 (named B2)

On the two faulty tunnels all is fine at the beginning. After a few days (1-14 days) some networks stop working. So I can ping from both networks A1 and A2 the network B1, but only from A2 the network B2. Pings from A1 to B2 will time out. The ASA on site A shows tx=0 traffic for A1 <=> B2, but rx traffic counts up. On ASA B it shows rx=0 for B2<=>A1 and tx counts up.

This happens unexpected after different periods. Sometimes it hits ASA on site B, where tx=0, sometimes it is ASA on site A.

I tried to fix it following commands:

clear crypto isakmp sa
clear crypto ipsec sa
clear xlate

but nothing worked. The only solution at the moment is to reboot the ASA where the tx count shows 0. After reboot all is fine for a while.

On one of the affected sites we have an ASA-Failover configuration. A switchover of the active appliance also solves the issue. But if you switch back prior rebooting the former primary the issue will return immediatelly.

I think it is no configuration because:

- All tunnels are configured the same way, and one of them is running for moths without any issue

- The tunnels are working for all subnet combinations after a reboot

- The issue occurs after different and long time periods. So I think the period between to failures is to long to be caused by tunnel timeouts a.s.o.

 

All ASA are running 9.1.(5).21.

I upgraded the firmware to several releases the last months, and had the same issue with any release I tested.

 

So I hope someone else had also this problem and found a solution.

1 Accepted Solution

Accepted Solutions

Jay Brown
Level 1
Level 1

Hey Christian!

Any luck resolving this, or finding the root cause?

Thanks

View solution in original post

7 Replies 7

Jay Brown
Level 1
Level 1

Hey Christian!

Any luck resolving this, or finding the root cause?

Thanks

Hey Jay,

I am sorry, of course your reply is not the correct answer. I mistakly pressed the button.

I still have not found out the root cause.

I was having the same issue (ASA5510 v8.4(2)), and the "clear crypro ipsec sa inactive" worked for us. The next step for me would have been a reload of the ASA.

The issue might be described in the following link https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsu50237

Diego Lopez
Level 1
Level 1

Based on the explanation that on one side you see decaps but not encaps and the other just encaps looks like a duplicate SPI the ASA might be using the wrong SPI to encrypt the traffic. I will agree that this might not be a configuration issue because the ASA is just randomly just not encrypting traffic and after a reload it works. The workaround for this kind of issues based on this bug is a reload:

https://tools.cisco.com/bugsearch/bug/CSCsu50237

You can check what SPI is in use on the ASA that is showing 0 encaps when this issue is occurring. First run a packet tracer for the traffic that stop working something like this:

packet-tracer input inside icmp 192.168.50.5 8 0 192.168.61.5 de

look for the VPN phase

Phase: 10


Type: VPN
Subtype: encrypt
Subtype:
Result: ALLOW
Config:

Additional Information:
Forward Flow based lookup yields rule:
out id=0x3c01878, priority=70, domain=encrypt, deny=false

hits=374, user_data=0x02d5re0e5, cs_id=0x0, flags=0x4000, protocol=0

src ip=192.168.50.0, mask=255.255.255.0, port=0

dst ip=192.168.61.0, mask=255.255.255.0, port=0, dscp=0x0

This is the current SPI 0x02d5re0e5 that is used to encrypt the traffic based on the packet tracer.

Now need to look for the outbound SPI that is in use in the IPsec SA for that particular traffic

"sh crypto ipsec sa peer aa.xx.yy.zz"

peer address: aa.xx.yy.zz
    Crypto map tag: xxxx, seq num: 1, local addr:

      access-list cryptomap extended permit ip 192.168.50.0 255.255.255.0 192.168.61.0 255.255.255.0
      local ident (addr/mask/prot/port): (192.168.50.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.61.0/255.255.255.0/0/0)
      current_peer: aa.xx.yy.zz

      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 480, #pkts decrypt: 480, #pkts verify:480 0
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: , remote crypto endpt.:

      path mtu 1500, ipsec overhead 58, media mtu 1500


    inbound esp sas:
      spi: 0xB2E9E500 (3001672960)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 100327424, crypto-map: xxxxx
         sa timing: remaining key lifetime (kB/sec): (4374000/1598)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
   

outbound esp sas:
      spi: 2a13e894 (957567060)
         transform: esp-3des esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, }
         slot: 0, conn_id: 100327424, crypto-map: xxxxx
         sa timing: remaining key lifetime (kB/sec): (4373976/1598)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

with this we can see that SPI 02d5re0e5 is the current SPI but the SA is doing the encryption with 2a13e894 

A command "clear crypto ipsec sa inactive" was introduced starting 9.1 you can try it, this should delete the duplicate SPI entry from the asp table, but some times it doesn't work! 

if this doesn't help then need to reload the device every time this happens. No other solution available at the moment.

Hello Diego,

thanks for your reply.

I checked the value of the user_data field against the SPI for working and for faulty connections. But in both cases the user_data value does not match the SPI of the connection.

The "clear crypro ipsec sa inactive" command deletes some SPIs but does not solve the problem.

At the moment the only option is to reboot the ASA, which is a very unsatisfying solution in a productive environment.

But I found another workaround without rebooting the device:

1. Make some changes in the affected Crypto Map, e.g. add a dummy host object.

2. Apply changes; ASA builds a new tunnel to same remote peer

3. At StS Monitoring select the old tunnel (can be identified by the uptime of the tunnel) and press the Logout button

4. Undo the changes in the affected Crypto Map

5. Apply changes; ASA builds a new tunnel to the remote peer again.

6. At StS Monitoring select again the older tunnel and press Logout.

7. Save changes to memory

Now the tunnel wokrs correctly.

Regards Christian

Hi Everyone,

Does this mean that when we do packet tracer and show cry ipsec sa peer the SPI value should be same

in both ?

 

Regards

Mahesh

Wai wai
Level 1
Level 1

Hi Christian,

This was a long time ago but had you find out the root cause already?

I had a similar issue with you exactly. The IPsec tunnels are all working fine for a very long time but sometimes the IPsec VPN will have this issue as you mentioned. 

The only difference between you and us is our main site(HQ) is running on ASA software version 9.2(2).4, HA Active-Passive mode, and all of our other site offices are running with other products (such as FortiGate firewall and etc).

When the issue happened for an IPsec VPN tunnel connection between the HQ and site office, it is not affecting the whole IPsec VPN tunnel but it is affected a single network subnet of the IPsec VPN tunnel only. The status of the IPsec VPN tunnel is still showing status up on both ends.

"clear crypto isakmp sa" or "clear crypto ipsec sa" will not work

However, reboot the ASA or force the failover to the passive ASA unit will solve the issue and the affected IPsec VPN tunnel connection will be restored for the affected network subnet.

Luckily this issue is not happening frequently to us so we still can tolerate it.