Re: IPSec routing halts randomly

kyiver_voip · ‎09-05-2011

Hello,

We have got an issue with an IPSec tunnel between Cisco PIX515E and Juniper firewall, the last one is managed by ISP. The tunnel is set up over the Internet.

There are a number of subnets running via this tunnel. The issue is that sometimes the connectivity between some of the subnet halts. So the users (and nagios) would report they can not access the service over the tunnel, while I would access the PIX over the tunnel by telnet just fine (from another side), then issue 'clear ipsec sa', and the connectivity would be restored. This happens randomly, one or few times a day. This is very urgent, so your help would be much appreciated.

Many thanks.

Andri

kyiver_voip · ‎09-05-2011

I have checked the MRTG for traffic statistics. It seems that occurences of IPSec routing issues match the traffic spikes on the pix outside interface...

There are video calls going throught the tunnel. And unfortunately it seems that the tunnel stops working exactly when it's needed the most...

adrian.lesovici · ‎09-05-2011

Hi Andri,

I have the same problem here with 2 Cisco ASA, both version 8.41.

Only one SA from 5 protected by this tunnel is having problem.

fw# sh ipsec sa spi 83813154

spi: 0x83813154

Crypto map tag: outside_map, seq num: 100, local addr: 194.a.a.a

access-list outside_cryptomap_100 extended permit ip 172.31.22.0 255.255.255.0 172.30.10.0 255.255.255.0

local ident (addr/mask/prot/port): (172.31.22.0/255.255.255.0/0/0)

remote ident (addr/mask/prot/port): (172.30.10.0/255.255.255.0/0/0)

current_peer: 195.b.b.b

#pkts encaps: 24817511, #pkts encrypt: 24810950, #pkts digest: 24810950

#pkts decaps: 17000097, #pkts decrypt: 17000097, #pkts verify: 17000097

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 24817511, #pkts comp failed: 0, #pkts decomp failed: 0

#pre-frag successes: 12, #pre-frag failures: 0, #fragments created: 24

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 36

#send errors: 6573, #recv errors: 0

local crypto endpt.: 194.a.a.a/0, remote crypto endpt.: 195.b.b.b/0

path mtu 1500, ipsec overhead 58, media mtu 1500

current outbound spi: 83813154

current inbound spi : 25F7698F

inbound esp sas:

spi: 0x25F7698F (636971407)

transform: esp-3des esp-sha-hmac no compression

in use settings ={L2L, Tunnel, PFS Group 2, }

slot: 0, conn_id: 12410880, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (3688912/19958)

IV size: 8 bytes

replay detection support: Y

Anti replay bitmap:

0xFFFFFFFF 0xFFFFFFFF

outbound esp sas:

spi: 0x83813154 (2206282068)

transform: esp-3des esp-sha-hmac no compression

in use settings ={L2L, Tunnel, PFS Group 2, }

slot: 0, conn_id: 12410880, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (0/19958)

IV size: 8 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001

adrian.lesovici · ‎09-05-2011

and here is some syslogs:

Aug 31 16:39:08 172.31.22.1 %ASA-5-713041: Group = 195.b.b.b, IP = 195.b.b.b, IKE Initiator: Rekeying Phase 2, Intf outside, IKE Peer 195.b.b.b local Proxy Address 172.31.22.0, remote Proxy Address 172.30.10.0, Crypto map (outside_map)

Aug 31 16:39:08 172.31.22.1 %ASA-6-602303: IPSEC: An outbound LAN-to-LAN SA (SPI= 0x83813154) between 194.a.a.a and 195.b.b.b (user= 195.b.b.b) has been created.

Aug 31 16:39:08 172.31.22.1 %ASA-5-713049: Group = 195.b.b.b, IP = 195.b.b.b, Security negotiation complete for LAN-to-LAN Group (195.b.b.b) Initiator, Inbound SPI = 0x25f7698f, Outbound SPI = 0x83813154

Aug 31 16:39:08 172.31.22.1 %ASA-6-602303: IPSEC: An inbound LAN-to-LAN SA (SPI= 0x25F7698F) between 194.a.a.a and 195.b.b.b (user= 195.b.b.b) has been created.

Aug 31 16:39:08 172.31.22.1 %ASA-5-713120: Group = 195.b.b.b, IP = 195.b.b.b, PHASE 2 COMPLETED (msgid=742e8ca0)

Aug 31 16:39:08 172.31.22.1 %ASA-6-602304: IPSEC: An outbound LAN-to-LAN SA (SPI= 0x045A1A7A) between 194.a.a.a and 195.b.b.b (user= 195.b.b.b) has been deleted.

Aug 31 16:39:08 172.31.22.1 %ASA-6-602304: IPSEC: An inbound LAN-to-LAN SA (SPI= 0x4D1E80F6) between 195.b.b.b and 194.a.a.a (user= 195.b.b.b) has been deleted.

Aug 31 16:39:08 172.30.10.1 %ASA-6-602303: IPSEC: An outbound LAN-to-LAN SA (SPI= 0x25F7698F) between 195.b.b.b and 194.a.a.a (user= 194.a.a.a) has been created.

Aug 31 16:39:08 172.30.10.1 %ASA-5-713049: Group = 194.a.a.a, IP = 194.a.a.a, Security negotiation complete for LAN-to-LAN Group (194.a.a.a) Responder, Inbound SPI = 0x83813154, Outbound SPI = 0x25f7698f

Aug 31 16:39:08 172.30.10.1 %ASA-6-602303: IPSEC: An inbound LAN-to-LAN SA (SPI= 0x83813154) between 195.b.b.b and 194.a.a.a (user= 194.a.a.a) has been created.

Aug 31 16:39:08 172.30.10.1 %ASA-6-602304: IPSEC: An outbound LAN-to-LAN SA (SPI= 0x4D1E80F6) between 195.b.b.b and 194.a.a.a (user= 194.a.a.a) has been deleted.

Aug 31 16:39:08 172.30.10.1 %ASA-6-602304: IPSEC: An inbound LAN-to-LAN SA (SPI= 0x045A1A7A) between 194.a.a.a and 195.b.b.b (user= 194.a.a.a) has been deleted.

kyiver_voip · ‎09-08-2011

Hi Adrian,

From the debugs I found lots of the following messages when the issue happen:

debug cry ipsec 127

debug cry isa 127

%PIX-4-402119: IPSEC: Received an ESP packet (SPI= 0x25C9CB7C, sequence number= 0x116EEE) from 178.X.X.X (user= 178.X.X.X) to 194.X.X.X that failed anti-replay checking.

%PIX-4-402119: IPSEC: Received an ESP packet (SPI= 0x25C9CB7C, sequence number= 0x116EED) from 178.X.X.X (user= 178.X.X.X) to 194.X.X.X that

So I have disabled Ipsec Anti Replay Window (

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t14/feature/guide/gt_iarwe.html

), and we haven't had this issue again, at least for more than a day now. I hope this fix will resolve the problem.

Also we adjusted the following parameters to increase SA lifetime:

crypto ipsec security-association lifetime kilobytes 32608000

crypto ipsec security-association lifetime seconds 28800

I hope this helps!

Andri

adrian.lesovici · ‎09-08-2011

Hi Andri,

I too have the tunnel up for 2 days.

I did 2 things: One is that I switched to IKEv2 with and the other is that I deleted the second IP Address of the peer in the crypto map.

I will set the second IP again because I want failover for the VPN but I will wait for a couple of days. I'll get back with news.

Adrian

kyiver_voip · ‎09-08-2011

Thanks Adrian. I will also keep you updated. (btw, the tunnel is still up )