VPN Tunnel Dropping

adamtodd16 · ‎01-08-2013

We have approx. 40 branch offices that connect to our core IOS Firewall (2951) over ipsec VPN Tunnel. One particular site has been facing issues over the past few days. This site will sporadically drop it's VPN Tunnel and reestablish after a few seconds.

If I run debug crypto ipsec and crypto isakmp on the site that is dropping, it is constantly going through the DPD process. If I run these same commands on another site, they seem to run DPD at all.

Here is some of the output I am seeing on the site that is failing.

Any help would be greatly appreciated.

Jan 8 11:18:38.873 AST: %FW-6-DROP_PKT: Dropping tcp session 111.222.3.106:50083 96.16.47.144:80 due to Stray Segment with ip ident 54856 tcpflags 0x5004 seq.no 2154004347 ack 0

Jan 8 11:18:46.061 AST: ISAKMP (4028): received packet from 111.222.255.106 dport 500 sport 500 Global (I) QM_IDLE

Jan 8 11:18:46.061 AST: ISAKMP: set new node -1497488895 to QM_IDLE

Jan 8 11:18:46.061 AST: ISAKMP:(4028): processing HASH payload. message ID = 2797478401

Jan 8 11:18:46.061 AST: ISAKMP:(4028): processing SA payload. message ID = 2797478401

Jan 8 11:18:46.061 AST: ISAKMP:(4028):Checking IPSec proposal 1

Jan 8 11:18:46.061 AST: ISAKMP: transform 1, ESP_AES

Jan 8 11:18:46.061 AST: ISAKMP: attributes in transform:

Jan 8 11:18:46.061 AST: ISAKMP: encaps is 1 (Tunnel)

Jan 8 11:18:46.061 AST: ISAKMP: SA life type in seconds

Jan 8 11:18:46.061 AST: ISAKMP: SA life duration (basic) of 3600

Jan 8 11:18:46.061 AST: ISAKMP: SA life type in kilobytes

Jan 8 11:18:46.065 AST: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0

Jan 8 11:18:46.065 AST: ISAKMP: authenticator is HMAC-SHA

Jan 8 11:18:46.065 AST: ISAKMP: key length is 128

Jan 8 11:18:46.065 AST: ISAKMP:(4028):atts are acceptable.

Jan 8 11:18:46.065 AST: IPSEC(validate_proposal_request): proposal part #1

Jan 8 11:18:46.065 AST: IPSEC(validate_proposal_request): proposal part #1,

(key eng. msg.) INBOUND local= 111.222.3.106:0, remote= 111.222.255.106:0,

local_proxy= 192.168.20.0/255.255.255.0/0/0 (type=4),

remote_proxy= 192.168.0.0/255.255.0.0/0/0 (type=4),

protocol= ESP, transform= NONE (Tunnel),

lifedur= 0s and 0kb,

spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0

Jan 8 11:18:46.065 AST: Crypto mapdb : proxy_match

src addr : 192.168.20.0

dst addr : 192.168.0.0

protocol : 0

src port : 0

dst port : 0

Jan 8 11:18:46.069 AST: ISAKMP:(4028): processing NONCE payload. message ID = 2797478401

Jan 8 11:18:46.069 AST: ISAKMP:(4028): processing ID payload. message ID = 2797478401

Jan 8 11:18:46.069 AST: ISAKMP:(4028):QM Responder gets spi

Jan 8 11:18:46.069 AST: ISAKMP:(4028):Node 2797478401, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH

Jan 8 11:18:46.069 AST: ISAKMP:(4028):Old State = IKE_QM_READY New State = IKE_QM_SPI_STARVE

Jan 8 11:18:46.081 AST: ISAKMP:(4028): Creating IPSec SAs

Jan 8 11:18:46.081 AST: inbound SA from 111.222.255.106 to 111.222.3.106 (f/i) 0/ 0

(proxy 192.168.0.0 to 192.168.20.0)

Jan 8 11:18:46.081 AST: has spi 0x50B3B8D5 and conn_id 0

Jan 8 11:18:46.081 AST: lifetime of 3600 seconds

Jan 8 11:18:46.081 AST: lifetime of 4608000 kilobytes

Jan 8 11:18:46.081 AST: outbound SA from 111.222.3.106 to 111.222.255.106 (f/i) 0/0

(proxy 192.168.20.0 to 192.168.0.0)

Jan 8 11:18:46.081 AST: has spi 0xB7A278EA and conn_id 0

Jan 8 11:18:46.081 AST: lifetime of 3600 seconds

Jan 8 11:18:46.081 AST: lifetime of 4608000 kilobytes

Jan 8 11:18:46.085 AST: IPSEC(key_engine): got a queue event with 1 KMI message(s)

Jan 8 11:18:46.085 AST: Crypto mapdb : proxy_match

src addr : 192.168.20.0

dst addr : 192.168.0.0

protocol : 0

src port : 0

dst port : 0

Jan 8 11:18:46.085 AST: IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer 111.222.255.106

Jan 8 11:18:46.085 AST: IPSEC(create_sa): sa created,

(sa) sa_dest= 111.222.3.106, sa_proto= 50,

sa_spi= 0x50B3B8D5(1353955541),

sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 1257

sa_lifetime(k/sec)= (4522087/3600)

Jan 8 11:18:46.085 AST: IPSEC(create_sa): sa created,

(sa) sa_dest= 111.222.255.106, sa_proto= 50,

sa_spi= 0xB7A278EA(3080878314),

sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 1258

sa_lifetime(k/sec)= (4522087/3600)

Jan 8 11:18:46.085 AST: ISAKMP:(4028): sending packet to 111.222.255.106 my_port 500 peer_port 500 (I) QM_IDLE

Jan 8 11:18:46.085 AST: ISAKMP:(4028):Sending an IKE IPv4 Packet.

Jan 8 11:18:46.089 AST: ISAKMP:(4028):Node 2797478401, Input = IKE_MESG_INTERNAL, IKE_GOT_SPI

Jan 8 11:18:46.089 AST: ISAKMP:(4028):Old State = IKE_QM_SPI_STARVE New State = IKE_QM_R_QM2

Jan 8 11:18:46.093 AST: ISAKMP (4028): received packet from 111.222.255.106 dport 500 sport 500 Global (I) QM_IDLE

Jan 8 11:18:46.097 AST: ISAKMP:(4028):deleting node -1497488895 error FALSE reason "QM done (await)"

Jan 8 11:18:46.097 AST: ISAKMP:(4028):Node 2797478401, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH

Jan 8 11:18:46.097 AST: ISAKMP:(4028):Old State = IKE_QM_R_QM2 New State = IKE_QM_PHASE2_COMPLETE

Jan 8 11:18:46.097 AST: IPSEC(key_engine): got a queue event with 1 KMI message(s)

Jan 8 11:18:46.101 AST: IPSEC(key_engine_enable_outbound): rec'd enable notify from ISAKMP

Jan 8 11:18:46.101 AST: IPSEC(key_engine_enable_outbound): enable SA with spi 3080878314/50

Jan 8 11:18:46.101 AST: IPSEC(update_current_outbound_sa): get enable SA peer 111.222.255.106 current outbound sa to SPI B7A278EA

Jan 8 11:18:46.101 AST: IPSEC(update_current_outbound_sa): updated peer 111.222.255.106 current outbound sa to SPI B7A278EA

ALIAOF_ · ‎01-08-2013

Can you share the config from the site you are having issues with?

adamtodd16 · ‎01-10-2013

Sorry for the late reply.

----------------------------------------

!

! Last configuration change at 12:55:11 AST Tue Jan 8 2013 by rtradmin

! NVRAM config last updated at 12:55:13 AST Tue Jan 8 2013 by rtradmin

version 15.1

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

!

logging buffered 51000

no logging console

!

no aaa new-model

!

clock timezone AST -4 0

clock summer-time AST recurring

network-clock-participate wic 1

network-clock-select 1 T1 0/1/0

crypto pki token default removal timeout 0

!

no ip source-route

no ip gratuitous-arps

!

no ip dhcp use vrf connected

ip dhcp excluded-address 172.20.20.1 172.20.20.3

!

ip dhcp pool GUEST

network 172.20.20.0 255.255.255.0

default-router 172.20.20.1

dns-server 10.1.200.50

lease 0 1

!

ip cef

ip flow-cache timeout active 1

no ip domain lookup

ip inspect log drop-pkt

ip inspect one-minute high 1000

ip inspect one-minute low 800

ip inspect tcp max-incomplete host 150 block-time 0

ip inspect name FIREWALL dns

ip inspect name FIREWALL udp

ip inspect name FIREWALL ftp

ip inspect name FIREWALL fragment maximum 256 timeout 1

ip inspect name FIREWALL ntp

ip inspect name FIREWALL pptp

ip inspect name FIREWALL skinny

ip inspect name FIREWALL icmp router-traffic

ip inspect name FIREWALL tcp

no ipv6 cef

!

multilink bundle-name authenticated

!

parameter-map type inspect global

log dropped-packets enable

isdn switch-type primary-dms100

!

voice rtp send-recv

!

voice service voip

fax protocol t38 version 0 ls-redundancy 2 hs-redundancy 0 fallback none

!

voice class h323 1

h225 timeout tcp establish 3

call preserve

!

voice translation-rule 1

rule 1 /\(.*\)/ /506694*/

!

voice-card 0

dsp services dspfarm

!

redundancy

!

controller T1 0/1/0

pri-group timeslots 1-10,24

!

ip tftp source-interface GigabitEthernet0/1.120

!

crypto isakmp policy 1

encr aes

authentication pre-share

group 2

crypto isakmp key C00keAqUa135 address 222.222.255.106 no-xauth

crypto isakmp invalid-spi-recovery

!

crypto ipsec transform-set AES128 esp-aes esp-sha-hmac

!

crypto map ADAM_VPN 10 ipsec-isakmp

set peer 222.222.255.106

set transform-set AES128

match address VPN-NETWORKS

!

interface GigabitEthernet0/0

description INTERNET

bandwidth 15000

ip address 222.222.3.106 255.255.255.248

ip access-group INBOUND in

no ip redirects

no ip unreachables

ip flow ingress

ip flow egress

ip nat outside

ip inspect FIREWALL out

ip virtual-reassembly in

ip verify unicast reverse-path

speed 100

full-duplex

no cdp enable

crypto map ADAM_VPN

!

interface Service-Engine0/1

ip unnumbered GigabitEthernet0/1.20

service-module ip address 192.168.20.254 255.255.255.0

service-module ip default-gateway 192.168.20.1

!

interface GigabitEthernet0/1

no ip address

ip tcp adjust-mss 1452

speed 100

full-duplex

!

interface GigabitEthernet0/1.20

encapsulation dot1Q 20

ip address 192.168.20.1 255.255.255.0

ip access-group OUTBOUND in

ip helper-address 10.10.20.11

ip nat inside

ip virtual-reassembly in

h323-gateway voip interface

h323-gateway voip bind srcaddr 192.168.20.1

!

interface GigabitEthernet0/1.120

encapsulation dot1Q 120

ip address 10.10.20.1 255.255.255.0

ip access-group OUTBOUND in

ip helper-address 10.10.20.11

ip nat inside

ip virtual-reassembly in

!

interface GigabitEthernet0/1.220

description GUEST NETWORK

encapsulation dot1Q 220

ip address 172.20.20.1 255.255.255.0

ip access-group GUEST in

ip nat inside

ip virtual-reassembly in

!

interface Serial0/1/0:23

no ip address

encapsulation hdlc

isdn switch-type primary-ni

isdn incoming-voice voice

no cdp enable

!

ip forward-protocol nd

!

ip flow-export version 5

ip flow-export destination 10.1.200.63 2055

ip flow-top-talkers

top 20

sort-by bytes

!

no ip http server

no ip http secure-server

ip nat inside source list NAT interface GigabitEthernet0/0 overload

ip route 0.0.0.0 0.0.0.0 222.222.3.105

ip route 192.168.20.254 255.255.255.255 Service-Engine0/1

!

ip access-list standard SNMP

permit 10.1.200.63

!

ip access-list extended DF

permit tcp any any

ip access-list extended GUEST

permit udp any any eq bootpc

permit udp any any eq bootps

permit udp 172.20.20.0 0.0.0.255 host 10.1.200.50 eq domain

deny ip 172.20.20.0 0.0.0.255 10.0.0.0 0.255.255.255

deny ip 172.20.20.0 0.0.0.255 192.168.0.0 0.0.255.255

permit ip any any

ip access-list extended INBOUND

permit udp any any eq snmp

permit udp host 222.222.255.106 host 222.222.51.34 eq isakmp

permit esp host 222.222.255.106 host 222.222.51.34

permit tcp host 222.222.3.83 host 222.222.51.34 eq 22

permit udp host 888.888.888.111 any eq isakmp

permit esp host 888.888.888.111 any

permit tcp host 222.222.3.82 host 222.222.51.34 eq 22

permit udp host 222.222.255.106 host 222.222.3.106 eq isakmp

permit esp host 222.222.255.106 host 222.222.3.106

permit udp host 142.176.0.220 host 222.222.3.106 eq isakmp

permit esp host 142.176.0.220 host 222.222.3.106

permit tcp host 205.174.163.163 host 222.222.3.106 eq 22

permit tcp host 156.34.144.14 host 222.222.3.106 eq 22

permit tcp host 222.222.3.82 host 222.222.3.106 eq 22

permit tcp host 156.34.144.2 host 222.222.3.106 eq 22

permit tcp host 222.222.3.83 host 222.222.3.106 eq 22

permit tcp host 222.222.3.94 host 222.222.3.106 eq 22

permit tcp host 222.222.255.106 host 222.222.3.106 eq 22

permit gre host 222.222.255.110 host 222.222.51.34

permit tcp host 216.155.75.44 host 222.222.3.106 eq 1723

permit gre host 216.155.75.44 host 222.222.3.106

permit gre host 222.222.255.110 host 222.222.3.106

permit gre host 151.204.177.194 host 222.222.51.34

permit gre host 151.204.177.194 host 222.222.3.106

permit gre host 216.57.221.5 host 222.222.51.34

permit gre host 216.57.221.5 host 222.222.3.106

permit gre host 72.164.207.42 host 222.222.51.34

permit gre host 72.164.207.42 host 222.222.3.106

permit gre host 64.122.171.90 host 222.222.51.34

permit gre host 64.122.171.90 host 222.222.3.106

permit icmp host 222.222.255.106 host 222.222.51.34

permit icmp host 222.222.255.106 host 222.222.3.106

permit tcp host 64.62.12.66 host 222.222.3.106 eq 1723

permit gre host 64.62.12.66 host 222.222.3.106

permit icmp any host 10.1.200.63

deny ip any any log-input

ip access-list extended NAT

deny ip any 10.0.0.0 0.255.255.255

deny ip any 192.168.0.0 0.0.255.255

permit ip 10.10.20.0 0.0.0.255 any

permit ip 192.168.20.0 0.0.0.255 any

permit ip 172.20.20.0 0.0.0.255 any

ip access-list extended OUTBOUND

deny udp any host 222.222.255.106 eq isakmp

deny udp any host 222.222.255.106 eq non500-isakmp

deny esp any host 222.222.255.106

permit ip any any

ip access-list extended VPN-NETWORKS

permit ip 10.10.20.0 0.0.0.255 10.0.0.0 0.255.255.255

permit ip 192.168.20.0 0.0.0.255 10.0.0.0 0.255.255.255

permit ip 10.10.20.0 0.0.0.255 192.168.0.0 0.0.255.255

permit ip 192.168.20.0 0.0.0.255 192.168.0.0 0.0.255.255

permit ip 172.20.20.0 0.0.0.255 10.0.0.0 0.255.255.255

!

ip sla 10

icmp-echo 10.1.1.1 source-interface GigabitEthernet0/1.120

frequency 5

ip sla schedule 10 life forever start-time now

ip sla 15

icmp-echo 8.8.8.8

frequency 5

ip sla schedule 15 life forever start-time now

access-list 150 permit ip any any

disable-eadi

!

route-map clear-df-bit permit 10

match ip address DF

set ip df 0

!

snmp-server community 3C4a5I6S7p8 RW SNMP

snmp-server trap-source GigabitEthernet0/1.120

!

control-plane

!

voice-port 0/1/0:23

!

mgcp profile default

!

sccp local GigabitEthernet0/1.20

sccp ccm 192.168.10.20 identifier 2 version 5.0.1

sccp ccm 192.168.111.20 identifier 1 version 5.0.1

sccp

!

sccp ccm group 1

bind interface GigabitEthernet0/1.20

associate ccm 1 priority 1

associate ccm 2 priority 2

associate profile 2 register MTP588D09BB2A10

associate profile 1 register CFB588D09BB2A10

!

dspfarm profile 1 conference

codec g711ulaw

codec g711alaw

codec g729ar8

codec g729abr8

codec g729r8

codec g729br8

maximum sessions 3

associate application SCCP

!

dial-peer voice 1 pots

destination-pattern 9[2-9]......

port 0/1/0:23

!

dial-peer voice 2 pots

destination-pattern 91[2-9]..[2-9]......

port 0/1/0:23

forward-digits 11

!

dial-peer voice 3 pots

destination-pattern 918[8,7,0,6][8,7,0,6].......

port 0/1/0:23

forward-digits 11

!

dial-peer voice 4 pots

destination-pattern 9[2-9]11

port 0/1/0:23

!

dial-peer voice 5 pots

destination-pattern 90[2-9]..[2-9]......

port 0/1/0:23

forward-digits 11

!

dial-peer voice 6 pots

destination-pattern 9011T

port 0/1/0:23

prefix 011

!

dial-peer voice 1000 voip

preference 1

destination-pattern [1-8]...

progress_ind setup enable 3

session target ipv4:192.168.111.20

voice-class h323 1

dtmf-relay h245-alphanumeric

codec g711ulaw

no vad

!

dial-peer voice 1001 voip

preference 2

destination-pattern [1-8]...

progress_ind setup enable 3

session target ipv4:192.168.10.20

dtmf-relay h245-alphanumeric

codec g711ulaw

no vad

!

dial-peer voice 7 pots

destination-pattern 90

port 0/1/0:23

prefix 0

!

dial-peer voice 8 pots

incoming called-number .

direct-inward-dial

port 0/1/0:23

!

dial-peer voice 1007 voip

preference 2

destination-pattern 0797

progress_ind setup enable 3

session target ipv4:192.168.10.20

dtmf-relay h245-alphanumeric

codec g711ulaw

no vad

!

call-manager-fallback

max-conferences 4 gain -6

transfer-system full-consult

ip source-address 192.168.20.1 port 2000

max-ephones 20

max-dn 20

dialplan-pattern 1 506694 extension-length 4

!

line con 0

login local

line aux 0

line 130

no activation-character

no exec

transport preferred none

transport input all

transport output all

line vty 0 4

password 7 1511021F0725

login local

transport input all

!

scheduler allocate 20000 1000

ntp source GigabitEthernet0/1.120

ntp master

ntp server 10.1.1.1 prefer

!

no inservice

!

end