VPN tunnel looks up but no ping

[Freddy Andersen]

Hi,

I had a pix that had two working tunnels going to one 5510 and one 5520. Today the VPN tunnel to our 5520 stopped working but if I do sh cry isa sa both tunnels have QM_IDLE as the state. (both ends) I tried to debug crypto isakmp 255 but all I get is PEER_REAPER_TIMER and no other output on the pix side.

I'm looking for commands or something to try that could help ...

[Jennifer Halim]

Please share the output of "show cry ipsec sa" from both ends.

[Freddy Andersen]

#### 5520 ####

    Crypto map tag: Outside_map, seq num: 20, local addr: 66.xx.xx.134

      access-list Outside_cryptomap_20 permit ip 10.21.30.0 255.255.254.0 10.3.185.0 255.255.255.0

      local ident (addr/mask/prot/port): (10.21.30.0/255.255.254.0/0/0)

      remote ident (addr/mask/prot/port): (10.3.185.0/255.255.255.0/0/0)

      current_peer: 66.xxx.xx.18

      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

      #pkts decaps: 5739, #pkts decrypt: 5739, #pkts verify: 5739

      #pkts compressed: 0, #pkts decompressed: 0

      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0

      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

      #send errors: 0, #recv errors: 0

      local crypto endpt.: 66.xx.xx.134, remote crypto endpt.: 66.xxx.xx.18

      path mtu 1500, ipsec overhead 58, media mtu 1500

      current outbound spi: 27A6B8F0

    inbound esp sas:

      spi: 0xCAD7D446 (3403142214)

         transform: esp-3des esp-md5-hmac no compression

         in use settings ={L2L, Tunnel, PFS Group 2, }

         slot: 0, conn_id: 16035840, crypto-map: Outside_map

         sa timing: remaining key lifetime (kB/sec): (4373597/25842)

         IV size: 8 bytes

         replay detection support: Y

         Anti replay bitmap:

          0xFFFFFFFF 0xFFFFFFFF

    outbound esp sas:

      spi: 0x27A6B8F0 (665237744)

         transform: esp-3des esp-md5-hmac no compression

         in use settings ={L2L, Tunnel, PFS Group 2, }

         slot: 0, conn_id: 16035840, crypto-map: Outside_map

         sa timing: remaining key lifetime (kB/sec): (4374000/25832)

         IV size: 8 bytes

         replay detection support: Y

         Anti replay bitmap:

          0x00000000 0x00000001

#### pix ####

   local  ident (addr/mask/prot/port): (office_range_inside/255.255.255.0/0/0)

   remote ident (addr/mask/prot/port): (10.21.30.0/255.255.254.0/0/0)

   current_peer: 66.xx.xx.134:500

     PERMIT, flags={origin_is_acl,}

    #pkts encaps: 7665, #pkts encrypt: 7665, #pkts digest 7665

    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0

    #pkts compressed: 0, #pkts decompressed: 0

    #pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress failed: 0

    #send errors 794, #recv errors 0

     local crypto endpt.: 66.xxx.xx.18, remote crypto endpt.: 66.xx.xx.134

     path mtu 1500, ipsec overhead 56, media mtu 1500

     current outbound spi: cad7d446

     inbound esp sas:

      spi: 0x27a6b8f0(665237744)

        transform: esp-3des esp-md5-hmac ,

        in use settings ={Tunnel, }

        slot: 0, conn id: 3, crypto map: to_colo

        sa timing: remaining key lifetime (k/sec): (4608000/24961)

        IV size: 8 bytes

        replay detection support: Y

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

      spi: 0xcad7d446(3403142214)

        transform: esp-3des esp-md5-hmac ,

        in use settings ={Tunnel, }

        slot: 0, conn id: 4, crypto map: to_colo

        sa timing: remaining key lifetime (k/sec): (4607515/24954)

        IV size: 8 bytes

        replay detection support: Y

     outbound ah sas:

     outbound pcp sas:

   local  ident (addr/mask/prot/port): (office_range_inside/255.255.255.0/0/0)

   remote ident (addr/mask/prot/port): (10.4.1.0/255.255.255.0/0/0)

   current_peer: 66.xx.xx.134:0

     PERMIT, flags={origin_is_acl,}

    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest 0

    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0

    #pkts compressed: 0, #pkts decompressed: 0

    #pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress failed: 0

    #send errors 0, #recv errors 0

     local crypto endpt.: 66.xxx.xx.18, remote crypto endpt.: 66.xx.xx.134

     path mtu 1500, ipsec overhead 0, media mtu 1500

     current outbound spi: 0

     inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:

     outbound pcp sas:

[Jennifer Halim]

Base on the above output, encaps increases on the PIX ends, but no decaps, and the opposite on ASA ends, ie: decaps increase and encaps is zero.

That means traffic is being sent from PIX towards the ASA end, however, either the traffic does not reach the

10.21.30.0/255.255.254.0 network, or that network is not replying or being blocked somewhere, or some routing error hence there is no encaps on the ASA end.

[Freddy Andersen]

What could cuse an issue like this since  this was working yesterday and just magaically stopped working today. I can still use my VPN client to access the 5520 connecting to the same network as the site-2-site.

If there is any information from the 5520 that would be helpful I can get that...

[Jennifer Halim]

Well, that is the useful information from the output of "show cry ipsec sa" showing exactly where it is failing.

You would need to further investigate back towards your LAN network and see where it's failing.

If there is no changes on the ASA, there might be changes at your other network devices.

If you can still use your VPN client, then possibly it doesn't know how to route back towards the remote LAN subnet (

10.3.185.0/24).

[Freddy Andersen]

I'm lost right now trying to find the solution for this problem... What other information would you need to troubleshoot this issue? Do you know of anyone that might be interested in looking deeper into this issue? (consulting)

[Freddy Andersen]

if I do icmp trace and ping from the inside of our pix side to the inside if the asa side I see the ping return but it never gets encapsulated...

ICMP echo request from outside:10.3.185.106 to inside:10.21.31.56 ID=21782 seq=12 len=56

ICMP echo reply from inside:10.21.31.56 to outside:10.3.185.106 ID=21782 seq=12 len=56

Looks like there is a bug in 8.2.1 that I think I'm hitting.

CSCtb53186 Duplicate ASP crypto table entry causes firewall to not   encrypt traffic

I found that a reboot will clear this up but is there another way?

[Jennifer Halim]

Do you mean you did a reload and that fixed the issue?

If that is the case, I would suggest upgrading to the latest version of 8.2.x.

[mikull.kiznozki]

Hey Freddy, did a reboot fix it for you mate? which code do you run? I seem to have a similar issue, but mine is realted to the ASA 5540 running 8.4(2) which is not natting my tunnel traffic even though my nat statements are perfect and the packet tracer output shows everything to be okay!

W E I R D!

*having sleepless nights*

[Freddy Andersen]

I will reload tonight. I'm using a 5520 8.2(1) looking to upgrade to 8.2(5). Is that easy btw? I have 2 5520 in active/active..

Sent from Cisco Technical Support iPad App

[Jennifer Halim]

Yup, pretty easy. You can pre-upload the software to both the ASA.

Then when you are ready to reload it, just change the "boot system" to the latest software, save the config, and reload.

Here is the steps for your reference for Active/Active failover:

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/admin_swconfig.html#wp1057555

[mikull.kiznozki]

let us know how you go mate. I am very reluctant to bounce my asa as it is in a multitenant environment and being accessed 24/7..a downtime would be $$$.. so much for managing and setting up a whole asa all by myself. i wish I had a team to discuss my problems with. lol

[Freddy Andersen]

Hi Guys,

This weekend I reloaded both active and standby asa but I'm still having the same issue with the one broken tunnel.

I tried re-creating the tunnel and that did not work I have used two pixes on the other end bit they both gives the same issue... Full connection BUT one side does encaps and the other decaps only!

My next step is to get 8.2(5) and do an upgrade this week.

I have other L2L tunnels working + I have about 10 User VPN connections going to this same asa. This is just crazy, everything looks good too me...

[Jennifer Halim]

It doesn't quite sound like an ASA issue if other tunnels are working and vpn client is also working, plus you have already reloaded the unit as well.

Please check that you have the correct route back for this particular remote LAN network.

This is exactly what Mikull experience on his network, and it ended up with switch failure causes incorrect routing, and traffic is not being routed back towards the ASA for traffic destined to a particular remote LAN network.