EZVPN Connection Down causes Black Hole while BGP to MPLS is established and updating

haydavis · ‎05-11-2015

I have a router connected to MPLS network for Primary access and EZVPN for back. MPLS and BGP look stable and routing table is stable. Telnet/Ping access thorugh MPLS to router is good but can not reach anything on the LAN side of the router. Not even next hope. But from the router everyting on LAN side is reachable. User on the LAN site are unable to get past the MPLS/VPN router.

Logs show repeted Connection down messages.

May 10 21:38:23: %CRYPTO-6-EZVPN_CONNECTION_DOWN: (Client) User= Group=guccirro-h-g1 Server_public_addr=

Services were restored as soon as crytpo ipsec client ezvpn crypto_back inside was removed from the Local LAN interface.

Any idea why this would black hole traffic?

!
version 15.2
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime
service timestamps log datetime localtime
service password-encryption
!
hostname <removed>pip-yc188-3264451
!
boot-start-marker
boot system flash0:c2900-universalk9-mz.SPA.152-4.M4.bin
boot system flash0:c2900-universalk9-mz.SPA.152-4.M6a.bin
boot-end-marker
!
!
<enable secret >
!
aaa new-model
!
!
aaa authentication login default group tacacs+ local
aaa authorization commands 15 default group tacacs+ if-authenticated
aaa accounting commands 15 default
action-type stop-only
group tacacs+
!
!
!
!
!
!
aaa session-id common
!
ip cef
!
!
!
!

!
!
!
!
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
license udi pid CISCO2921/K9 sn FTX1838AN6G
!
!
<username >
!
redundancy
!
!
!
!
!
no ip ftp passive
ip ftp source-interface Loopback0
<ip ftp username >
<ip ftppassword >
!
class-map match-any af43
match access-group name af43
class-map match-any af42
match access-group name af42
class-map match-any af41
match access-group name af41
class-map match-any ef
match access-group name ef
class-map match-any af21
match access-group name af21
class-map match-any af12
match access-group name af12
class-map match-any af31
match access-group name af31
class-map match-any af13
match access-group name af13
class-map match-any af32
match access-group name af32
class-map match-any af23
match access-group name af23
class-map match-any af33
match access-group name af33
class-map match-any af22
match access-group name af22
class-map match-any af11
match access-group name af11
class-map match-any be
match access-group name be
class-map match-any drop
match access-group name drop
class-map match-any besteffort
match ip dscp default
class-map match-any realtime
match ip dscp cs5 ef
class-map match-any priority
match ip dscp cs4 af41 af42 af43
class-map match-any cs1
match access-group name cs1
class-map match-any cs2
match access-group name cs2
class-map match-any cs3
match access-group name cs3
class-map match-any cs4
match access-group name cs4
class-map match-any cs5
match access-group name cs5
class-map match-any cs6
match access-group name cs6
class-map match-any cs7
match access-group name cs7
class-map match-any catch-marked
match ip dscp ef
match ip dscp cs5
match ip dscp cs4
match ip dscp af41
match ip dscp af42
match ip dscp af43
match ip dscp cs3
match ip dscp af31
match ip dscp af32
match ip dscp af33
match ip dscp cs6
match ip dscp cs7
match ip dscp cs2
match ip dscp af21
match ip dscp af22
match ip dscp af23
match ip dscp cs1
match ip dscp af11
match ip dscp af12
match ip dscp af13
class-map match-any missioncritical
match ip dscp cs3 af31 af32 af33 cs6 cs7
class-map match-any transactional
match ip dscp cs2 af21 af22 af23
class-map match-any general
match ip dscp af11 af12 af13
class-map match-any scavenger
match ip dscp cs1
!
policy-map etm-<removed>
class realtime
priority 160
police 160000 conform-action transmit exceed-action drop
class priority
bandwidth remaining percent 40
random-detect dscp-based
class missioncritical
bandwidth remaining percent 39
random-detect dscp-based
class transactional
bandwidth remaining percent 16
random-detect dscp-based
class general
bandwidth remaining percent 1
random-detect dscp-based
class class-default
bandwidth remaining percent 4
random-detect dscp-based
policy-map shape-etm-<removed>
class class-default
shape average 3400000
   service-policy etm-<removed>
policy-map mark
class catch-marked
class ef
set ip dscp ef
class cs5
set ip dscp cs5
class cs4
set ip dscp cs4
class af41
set ip dscp af41
class af42
set ip dscp af42
class af43
set ip dscp af43
class cs3
set ip dscp cs3
class af31
set ip dscp af31
class af32
set ip dscp af32
class af33
set ip dscp af33
class cs6
set ip dscp cs6
class cs7
set ip dscp cs7
class cs2
set ip dscp cs2
class af21
set ip dscp af21
class af22
set ip dscp af22
class af23
set ip dscp af23
class cs1
set ip dscp cs1
class af11
set ip dscp af11
class af12
set ip dscp af12
class af13
set ip dscp af13
class drop
drop
class be
set ip dscp default
!
!
crypto isakmp keepalive 90 periodic
crypto isakmp nat keepalive 10
!
!
!
!
!
!
crypto ipsec client ezvpn crypto_back
connect auto
group <removed>rro-h-g1 key <removed>
mode network-extension
peer <removed>
acl multi-subnet
flow restrict
virtual-interface 1
< username >
xauth userid mode local
!
!
!
!
!
!
interface Loopback0
description LOOPBACK ADDRESS FOR MANAGEMENT AND NTP
ip address 172.27.23.31 255.255.255.255
!
interface Loopback1
description Access Through SGRR Only
ip address 10.252.28.109 255.255.255.255
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description EDBK681M0001
bandwidth 4000
no ip address
duplex full
speed 100
service-policy output shape-etm-<removed>
!
interface GigabitEthernet0/0.42
bandwidth 4000
encapsulation dot1Q 42
ip address <removed> 255.255.255.252
!
interface GigabitEthernet0/1
description DSL_Circuit
ip address <removed> 255.255.255.248
ip access-group firewall in
no ip unreachables
ip flow ingress
ip flow egress
duplex full
speed 100
crypto ipsec client ezvpn crypto_back
!
interface GigabitEthernet0/2
description Customer LAN
ip address 172.27.23.149 255.255.255.254 secondary
ip address 10.54.135.249 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1360
duplex full
speed 100
crypto ipsec client ezvpn crypto_back inside
service-policy input mark
!
interface Virtual-Template1 type tunnel
ip unnumbered GigabitEthernet0/2
ip mtu 1400
tunnel mode ipsec ipv4
!
!
router bgp 493
bgp log-neighbor-changes
aggregate-address 10.54.135.0 255.255.255.0 summary-only
redistribute connected route-map CONNECTED-TO-BGP
redistribute static route-map STATIC-TO-BGP
neighbor <removed> remote-as 65000
neighbor <removed> description Verizon Business MPLS Circuit NYC9E02 TenGigE0/0/0/2.37
neighbor <removed> soft-reconfiguration inbound
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip route 4.2.2.1 255.255.255.255 <removed>
ip route 8.8.8.8 255.255.255.255 <removed>
ip route 10.52.135.0 255.255.255.0 GigabitEthernet0/2 10.54.135.250 tag 999
ip route 10.54.135.0 255.255.255.0 GigabitEthernet0/2 10.54.135.250 tag 999
ip route <removed> 255.255.255.255 <removed>
ip tacacs source-interface Loopback0
!
ip access-list extended af12
remark Web-Cache
permit tcp any eq 8080 any
ip access-list extended af21
remark FTP
permit tcp any eq ftp any
permit tcp any any eq ftp
permit tcp any eq ftp-data any
permit tcp any any eq ftp-data
remark Live Meeting file transfer
permit tcp any range 6891 6900 any
permit tcp any any range 6891 6901
ip access-list extended af22
remark MAIL
permit tcp any eq smtp any
permit tcp any any eq smtp
remark Notes RPC
permit tcp any eq 1352 any
permit tcp any any eq 1352
ip access-list extended af31
permit tcp any eq www any
permit tcp any any eq www
permit tcp any eq domain any
permit tcp any any eq domain
permit udp any eq domain any
permit udp any any eq domain
remark SIP
permit tcp any eq 5060 any
permit tcp any any eq 5060
permit udp any eq 5060 any
permit udp any any eq 5060
permit tcp any eq telnet any
permit tcp any any eq telnet
remark OCS SIP
permit tcp any eq 5063 any
permit tcp any any eq 5063
remark Live Meeting SIP
permit tcp any range 5060 5061 any
permit tcp any any range 5060 5061
ip access-list extended af32
remark Kerberos
permit tcp any eq 88 any
permit tcp any any eq 88
remark LDAP
permit tcp any eq 389 any
permit tcp any any eq 389
remark Windows SMP
permit tcp any eq 445 any
permit tcp any any eq 445
remark Citrix
permit tcp any eq 1494 any
permit tcp any any eq 1494
remark SQL
permit tcp any eq 1433 any
permit tcp any any eq 1433
remark Remote Desktop
permit tcp any eq 3389 any
permit tcp any any eq 3389
ip access-list extended af33
remark Kerberos
permit udp any eq 88 any
permit udp any any eq 88
remark LDAP
permit udp any eq 389 any
permit udp any any eq 389
remark Citrix
permit udp any eq 1604 any
permit udp any any eq 1604
ip access-list extended af41
remark OCS
permit tcp any eq 8057 any
permit tcp any any eq 8057
permit tcp any range 7100 7103 any
permit tcp any any range 7100 7103
permit tcp any range 5350 5353 any
permit tcp any any range 5350 5353
ip access-list extended af42
remark Live Meeting
permit tcp any eq 1503 any
permit tcp any any eq 1503
ip access-list extended ef
remark Voice
permit ip 10.55.28.128 0.0.0.31 any
ip access-list extended firewall
deny   ip 0.0.0.0 0.255.255.255 any
deny   ip 10.0.0.0 0.255.255.255 any
deny   ip 100.64.0.0 0.63.255.255 any
deny   ip 127.0.0.0 0.255.255.255 any
deny   ip 172.16.0.0 0.15.255.255 any
deny   ip 192.0.0.0 0.0.0.255 any
deny   ip 192.0.2.0 0.0.0.255 any
deny   ip 192.168.0.0 0.0.255.255 any
deny   ip 198.18.0.0 0.1.255.255 any
deny   ip 198.51.100.0 0.0.0.255 any
deny   ip 224.0.0.0 31.255.255.255 any
permit esp any any
permit udp any eq non500-isakmp any
permit udp any eq isakmp any
permit icmp any any echo-reply
ip access-list extended multi-subnet
permit ip 10.52.135.0 0.0.0.255 any
permit ip 10.54.135.0 0.0.0.255 any
permit ip host 172.27.23.31 any
permit ip host 172.27.23.148 any
permit ip host 10.252.28.109 any
!
ip sla responder
access-list 54 remark customer SNMP access
access-list 54 permit 10.5.24.0 0.0.1.255
access-list 54 permit 10.5.34.0 0.0.0.255
access-list 54 permit 10.3.0.0 0.0.3.255
access-list 54 remark <removed> SNMP access
access-list 66 deny   any
!
route-map CONNECTED-TO-BGP deny 10
match interface GigabitEthernet0/1 Loopback1
!
route-map CONNECTED-TO-BGP permit 20
!
route-map STATIC-TO-BGP permit 10
match tag 999
!
!
snmp-server view BLOCK iso included
snmp-server view BLOCK ipAddrEntry.*.*.*.*.* excluded
snmp-server view BLOCK ipAddrEntry.*.<removed> included
snmp-server view BLOCK ipAddrEntry.*.172.27.*.* included
snmp-server view BLOCK ipNetToMediaEntry.*.*.*.*.*.* excluded
snmp-server view BLOCK ipNetToMediaEntry.*.*.<removed> included
snmp-server view BLOCK ipNetToMediaEntry.*.*.172.27.*.* included
snmp-server view BLOCK atEntry.*.*.*.*.*.*.* excluded
snmp-server view BLOCK atEntry.*.*.*.<removed> included
snmp-server view BLOCK atEntry.*.*.*.172.27.*.* included
snmp-server view rtt-view sysUpTime included
snmp-server view rtt-view ciscoPingMIB included
snmp-server view rtt-view ciscoRttMonMIB included
<snmp-server community >
<snmp-server community >
<snmp-server community >
<snmp-server community >
<snmp-server community >
snmp-server ifindex persist
snmp-server trap-source Loopback0
snmp-server location <removed> -   160 N GULPH RD, KING OF PRUSSIA
snmp-server contact Verizon Business 800-256-9284
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps entity-sensor threshold
snmp-server enable traps config
snmp-server enable traps frame-relay multilink bundle-mismatch
snmp-server enable traps frame-relay
snmp-server enable traps frame-relay subif
<snmp-server host >
<snmp-server host >
tacacs-server timeout 20
tacacs-server directed-request
<tacacs-server key >
!
!
!
control-plane
!
!
privilege exec level 1 traceroute
privilege exec level 1 ping
privilege exec level 1 show ip accounting
privilege exec level 1 show ip interface brief
privilege exec level 1 show ip interface
privilege exec level 1 show ip
privilege exec level 1 show running-config
privilege exec level 1 show configuration
privilege exec level 1 show
privilege exec level 1 clear ip accounting
privilege exec level 1 clear ip
privilege exec level 1 clear
banner login ^CCCC

^C
!
line con 0
exec-timeout 5 0
<password >
line aux 0
exec-timeout 30 0
<password >
line 2
no activation-character
no exec
transport preferred none
transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
exec-timeout 30 0
<password >
transport input telnet
!
scheduler allocate 20000 1000
ntp update-calendar
!
end

patrikmilat · ‎11-18-2016

Seems you are hitting a bug similar to CSCux09048.

Workaround:remove flow restrict under crypto map