IPSec Tunnel drops randomly.

[Dave Wood]

Hi,

I have been trying to find out why our tunnel goes down every few hours. Sometime up for 24 hrs, sometimes couple hours.

Any suggestion or advice will be appreciated?

We have 3 sites

Site A 5505 8.2

Site B 5510 8.2

Site C 5520 9.1.6

Tunnel between Site A & B and A & C very stable. But tunnel between A & C not stable. Every couple hours it disconnects.

[Dave Wood]

We have 3 sites

Site A 5505 8.2

Site B 5510 8.2

Site C 5520 9.1.6

Tunnel between Site A & B and A & C very stable. But tunnel between A & C not stable. Every couple hours it disconnects.

************************************************
Site: B 0.0.0.90
************************************************
vpn# sh crypto isakmp sa

   Active SA: 2
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2

1   IKE Peer: 0.0.0.106
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE
2   IKE Peer: 0.0.0.27
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE

vpn# sh crypto isakmp

   Active SA: 2
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2

1   IKE Peer:0.0.0.106
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE
2   IKE Peer: 0.0.0.27
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE

Global IKE Statistics
Active Tunnels: 2
Previous Tunnels: 17
In Octets: 28304
In Packets: 210
In Drop Packets: 14
In Notifys: 30
In P2 Exchanges: 45
In P2 Exchange Invalids: 0
In P2 Exchange Rejects: 0
In P2 Sa Delete Requests: 9
Out Octets: 27912
Out Packets: 204
Out Drop Packets: 0
Out Notifys: 24
Out P2 Exchanges: 13
Out P2 Exchange Invalids: 0
Out P2 Exchange Rejects: 0
Out P2 Sa Delete Requests: 45
Initiator Tunnels: 11
Initiator Fails: 0
Responder Fails: 2
System Capacity Fails: 0
Auth Fails: 0
Decrypt Fails: 0
Hash Valid Fails: 0
No Sa Fails: 0

Global IPSec over TCP Statistics
--------------------------------
Embryonic connections: 0
Active connections: 0
Previous connections: 0
Inbound packets: 0
Inbound dropped packets: 0
Outbound packets: 0
Outbound dropped packets: 0
RST packets: 0
Recevied ACK heart-beat packets: 0
Bad headers: 0
Bad trailers: 0
Timer failures: 0
Checksum errors: 0
Internal errors: 0


************************************************
Site C: 0.0.0.106
************************************************
vpn3# sh crypto isakmp sa

IKEv1 SAs:

   Active SA: 2
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2

1   IKE Peer: 0.0.0.27
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE
2   IKE Peer: 0.0.0.90
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE

There are no IKEv2 SAs


vpn3# sh crypto isakmp

IKEv1 SAs:

   Active SA: 2
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2

1   IKE Peer: 0.0.0.27
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE
2   IKE Peer: 0.0.0.90
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE

There are no IKEv2 SAs

Global IKEv1 Statistics
  Active Tunnels:              2
  Previous Tunnels:          449
  In Octets:              468112
  In Packets:               3252
  In Drop Packets:           294
  In Notifys:               1164
  In P2 Exchanges:           299
  In P2 Exchange Invalids:     0
  In P2 Exchange Rejects:      0
  In P2 Sa Delete Requests:  297
  Out Octets:             448520
  Out Packets:              3153
  Out Drop Packets:            0
  Out Notifys:              1448
  Out P2 Exchanges:          174
  Out P2 Exchange Invalids:    0
  Out P2 Exchange Rejects:     0
  Out P2 Sa Delete Requests: 174
  Initiator Tunnels:         164
  Initiator Fails:             3
  Responder Fails:            14
  System Capacity Fails:       0
  Auth Fails:                  4
  Decrypt Fails:               0
  Hash Valid Fails:            0
  No Sa Fails:                 0

IKEV1 Call Admission Statistics
  Max In-Negotiation SAs:                150
  In-Negotiation SAs:                      0
  In-Negotiation SAs Highwater:            3
  In-Negotiation SAs Rejected:             0

Global IKEv2 Statistics
  Active Tunnels:                          0
  Previous Tunnels:                        0
  In Octets:                               0
  In Packets:                              0
  In Drop Packets:                         0
  In Drop Fragments:                       0
  In Notifys:                              0
  In P2 Exchange:                          0
  In P2 Exchange Invalids:                 0
  In P2 Exchange Rejects:                  0
  In IPSEC Delete:                         0
  In IKE Delete:                           0
  Out Octets:                              0
  Out Packets:                             0
  Out Drop Packets:                        0
  Out Drop Fragments:                      0
  Out Notifys:                             0
  Out P2 Exchange:                         0
  Out P2 Exchange Invalids:                0
  Out P2 Exchange Rejects:                 0
  Out IPSEC Delete:                        0
  Out IKE Delete:                          0
  SAs Locally Initiated:                   0
  SAs Locally Initiated Failed:            0
  SAs Remotely Initiated:                  0
  SAs Remotely Initiated Failed:           0
  System Capacity Failures:                0
  Authentication Failures:                 0
  Decrypt Failures:                        0
  Hash Failures:                           0
  Invalid SPI:                             0
  In Configs:                              0
  Out Configs:                             0
  In Configs Rejects:                      0
  Out Configs Rejects:                     0
  Previous Tunnels:                        0
  Previous Tunnels Wraps:                  0
  In DPD Messages:                         0
  Out DPD Messages:                        0
  Out NAT Keepalives:                      0
  IKE Rekey Locally Initiated:             0
  IKE Rekey Remotely Initiated:            0
  CHILD Rekey Locally Initiated:           0
  CHILD Rekey Remotely Initiated:          0

IKEV2 Call Admission Statistics
  Max Active SAs:                   No Limit
  Max In-Negotiation SAs:                752
  Cookie Challenge Threshold:          Never
  Active SAs:                              0
  In-Negotiation SAs:                      0
  Incoming Requests:                       0
  Incoming Requests Accepted:              0
  Incoming Requests Rejected:              0
  Outgoing Requests:                       0
  Outgoing Requests Accepted:              0
  Outgoing Requests Rejected:              0
  Rejected Requests:                       0
  Rejected Over Max SA limit:              0
  Rejected Low Resources:                  0
  Rejected Reboot In Progress:             0
  Cookie Challenges:                       0
  Cookie Challenges Passed:                0
  Cookie Challenges Failed:                0

Global IKEv1 IPSec over TCP Statistics
--------------------------------
Embryonic connections: 0
Active connections: 0
Previous connections: 0
Inbound packets: 0
Inbound dropped packets: 0
Outbound packets: 0
Outbound dropped packets: 0
RST packets: 0
Recevied ACK heart-beat packets: 0
Bad headers: 0
Bad trailers: 0
Timer failures: 0
Checksum errors: 0
Internal errors: 0

[Rahul Govindan]

I would recommend checking the syslogs and debugs during the time of the issue. The debugs that you would need to run are:

 debug crypto condition peer x.x.x.x

debug crypto isakmp 127

debug crypto ipsec 127

[Dave Wood]

Thank you Rahul.

I'm going to enable debug at the time of issue and post here.

[Dave Wood]

Rahul,

I turned on debug but during the time of the issue I saw only following log entries

Teardown ICMP connection for faddr .......

Built inbound ICMP connection for faddr.....

No crypto error or warning msg.

VPN statistics in the ASDM showing IPSEC tunnel is up. If I refresh, I can see Bytes Tx Rx changing.

If I logout tunnel from Site C, it will re-establish again.

I'm not sure where I'm missing...

[Rahul Govindan]

Maybe it is not the tunnel itself but traffic through the tunnel that is being affected. If the tunnel is not going down, try applying a capture on the inside on both sides of the tunnel to see what happens to the traffic that is affected. This will help understand what causes the traffic to be affected.