Core Router dropping interfaces every week

[Paul Smith]

Hi Guys,

I am administrating a Cisco 2109 router that is the core router and WAN router for 200 plus staff. Around once a week, certain tunnels will drop and I lose SSH to the WAN interface. Many staff lose internet access but not all. The only way to reboot is to connect to a PC and reboot the router from an internal interface. Logging is not giving away too much info. I have spoke to the ISP. Last week we changed both our interfaces to speed 1000 manually because the ISP tech was seeing mismatches. We thought that might fix the issue but no luck.I have supplied the running config below. Maybe someone can see some discrepancy and see why it would just break down once a week?

KDRHO-WAN-RTR-VPN01#sh run
Building configuration...

 
Current configuration : 10601 bytes
!
version 15.6
service timestamps debug datetime localtime
service timestamps log datetime localtime
no service password-encryption
!
hostname KDRHO-WAN-RTR-VPN01
!
boot-start-marker
boot-end-marker
!
!
logging count
logging userinfo
logging buffered 131000
!
no aaa new-model
ethernet lmi ce
clock timezone Bris 10 0
!
!
!
!
!
!
!
!
!
ip dhcp excluded-address 192.168.214.1 192.168.214.11
ip dhcp excluded-address 192.168.212.245 192.168.212.254
ip dhcp excluded-address 192.168.214.1 192.168.214.100
ip dhcp excluded-address 192.168.212.1 192.168.212.25
ip dhcp excluded-address 192.168.213.1 192.168.213.199
ip dhcp excluded-address 192.168.213.225 192.168.213.254
ip dhcp excluded-address 192.168.215.1 192.168.215.100
ip dhcp excluded-address 192.168.112.1 192.168.112.100
ip dhcp excluded-address 192.168.112.200 192.168.112.254
ip dhcp excluded-address 10.10.10.1
!
ip dhcp pool MGMTvlan
 network 192.168.213.0 255.255.255.0
 default-router 192.168.213.1
 dns-server 10.201.212.8 102.201.212.9 8.8.8.8
!
ip dhcp pool VOICE
 network 192.168.112.0 255.255.255.0
 default-router 192.168.112.1
 dns-server 10.201.212.8 10.201.212.9 8.8.8.8
 option 66 ascii http://172.30.100.70:5000/provisioning/3tpz9qkdr5
 option 42 ip 54.252.165.245
!
!
!
ip domain name kdr.local
ip name-server 8.8.8.8
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
license udi pid CISCO2901/K9 sn FGL2101105V
license boot module c2900 technology-package securityk9
!
!
username XXXXXXXXXXXXXXXXXXXXXX privilege 15 secret 5 XXXXXXXXXXXXXXXXX
username XXXXXXXXXXXXXXXXXXXXXX secret 5 XXXXXXXXXXXXXXXXXXXXXXXXX
username XXXXXXXXXXXXXXXXXXXXXX privilege 15 secret 5 XXXXXXXXXXXXXXXXXXXXXX
!
redundancy
!
crypto ikev2 proposal azure-proposal
 encryption aes-cbc-256 aes-cbc-128 3des
 integrity sha1
 group 2
crypto ikev2 proposal azure-proposal_2
 encryption aes-cbc-256 aes-cbc-128 3des
 integrity sha1
 group 2
!
crypto ikev2 policy azure-policy
 proposal azure-proposal
crypto ikev2 policy azure-policy_2
 proposal azure-proposal_2
!
crypto ikev2 keyring azure-keyring_2
 peer 52.237.XXX.XXX
  address 52.237.XXX.XXX
  pre-shared-key XXXXXXXXXXXXXXXXXXXXXX
 !
!
crypto ikev2 keyring azure-keyring
 peer 13.75.XXX.XXX
  address 13.75.XXX.XXX
  pre-shared-key XXXXXXXXXXXXXXXXXXXXXX
 !
!
!
crypto ikev2 profile azure-profile_2
 match address local interface GigabitEthernet0/1
 match identity remote address 52.237.XXX.XXX 255.255.255.255
 authentication local pre-share
 authentication remote pre-share
 keyring local azure-keyring_2
!
crypto ikev2 profile azure-profile
 match address local interface GigabitEthernet0/1
 match identity remote address 13.75.XXX.XXX 255.255.255.255
 authentication local pre-share
 authentication remote pre-share
 keyring local azure-keyring
!
!
!
track 100 ip sla 100 reachability
!
!
crypto keyring keyring-vpn-701df919-1  
  local-address 139.130.XXX.XXX
  pre-shared-key address 52.64.XXX.XXX key XXXXXXXXXXXXXXXXXXXXXX
crypto keyring test_keyring  
  local-address 139.130.XXX.XXX
  pre-shared-key address 122.149.XXX.XXX key XXXXXXXXXXXXXXXXXXXXXX
crypto keyring keyring-vpn-701df919-0  
  local-address 139.130.XXX.XXX
  pre-shared-key address 13.55.XXX.XXX key XXXXXXXXXXXXXXXXXXXXXX
!
crypto isakmp policy 200
 encr aes
 authentication pre-share
 group 2
 lifetime 28800
crypto isakmp keepalive 10 10
crypto isakmp profile isakmp-vpn-701df919-0
   keyring keyring-vpn-701df919-0
   match identity address 13.55.XXX.XXX 255.255.255.255
   local-address 139.130.XXX.XXX
crypto isakmp profile isakmp-vpn-701df919-1
   keyring keyring-vpn-701df919-1
   match identity address 52.64.XXX.XXX 255.255.255.255
   local-address 139.130.XXX.XXX
crypto isakmp profile test_vpn
   keyring test_keyring
   match identity address 122.149.XXX.XXX 255.255.255.255
   local-address 139.130.XXX.XXX
!
crypto ipsec security-association replay window-size 128
!
crypto ipsec transform-set azure-ipsec-proposal-set esp-aes 256 esp-sha-hmac
 mode tunnel
crypto ipsec transform-set ipsec-prop-vpn-701df919-0 esp-aes esp-sha-hmac
 mode tunnel
crypto ipsec transform-set ipsec-prop-vpn-701df919-1 esp-aes esp-sha-hmac
 mode tunnel
crypto ipsec transform-set azure-ipsec-proposal-set_2 esp-aes 256 esp-sha-hmac
 mode tunnel
crypto ipsec transform-set test_transform-set esp-aes esp-sha-hmac
 mode tunnel
crypto ipsec df-bit clear
!
!
crypto ipsec profile ipsec-vpn-701df919-0
 set transform-set ipsec-prop-vpn-701df919-0
 set pfs group2
!
crypto ipsec profile ipsec-vpn-701df919-1
 set transform-set ipsec-prop-vpn-701df919-1
 set pfs group2
!
crypto ipsec profile test_profile
 set transform-set test_transform-set
 set pfs group2
!
crypto ipsec profile vti
 set transform-set azure-ipsec-proposal-set
 set ikev2-profile azure-profile
!
crypto ipsec profile vti_2
 set transform-set azure-ipsec-proposal-set_2
 set ikev2-profile azure-profile_2
!
!
!
!
!
!
interface Tunnel1
 description Azure_VPN_KDRGC
 ip address 169.254.0.1 255.255.255.0
 ip tcp adjust-mss 1350
 tunnel source GigabitEthernet0/1
 tunnel mode ipsec ipv4
 tunnel destination 13.75.XXX.XXX
 tunnel protection ipsec profile vti
!
interface Tunnel2
 description AWS_VPN1
 ip address 169.254.32.82 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1379
 tunnel source 139.130.XXX.XXX
 tunnel mode ipsec ipv4
 tunnel destination 13.55.XXX.XXX
 tunnel protection ipsec profile ipsec-vpn-701df919-0
!
interface Tunnel3
 description AWS_VPN2
 ip address 169.254.33.2 255.255.255.252
 ip virtual-reassembly in
 ip tcp adjust-mss 1379
 tunnel source 139.130.XXX.XXX
 tunnel mode ipsec ipv4
 tunnel destination 52.64.XXX.XXX
 tunnel protection ipsec profile ipsec-vpn-701df919-1
!
interface Tunnel4
 ip address 169.254.40.1 255.255.255.252
 ip mtu 1398
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1358
 tunnel source 139.130.XXX.XXX
 tunnel destination 122.149.XXX.XXX
 tunnel protection ipsec profile test_profile
!
interface Tunnel5
 description Azure_VPN_KDCORP
 ip address 169.254.1.1 255.255.255.0
 ip tcp adjust-mss 1350
 tunnel source GigabitEthernet0/1
 tunnel mode ipsec ipv4
 tunnel destination 52.237.XXX.XXX
 tunnel protection ipsec profile vti_2
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 no ip address
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface GigabitEthernet0/0.10
 description CORPORATE LAN
 encapsulation dot1Q 10
 ip address 192.168.212.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
interface GigabitEthernet0/0.20
 description MOBILES
 encapsulation dot1Q 20
 ip address 192.168.215.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
interface GigabitEthernet0/0.50
 description GUEST
 encapsulation dot1Q 50
 ip address 192.168.214.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
interface GigabitEthernet0/0.99
 description MANAGEMENT VLAN
 encapsulation dot1Q 99
 ip address 192.168.213.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!         
interface GigabitEthernet0/0.100
 description VOICE
 encapsulation dot1Q 100
 ip address 192.168.112.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
interface GigabitEthernet0/0.125
 description LINK to KDR-WAN-RTR-FIBRE
 encapsulation dot1Q 125
 ip address 10.10.10.1 255.255.255.248
 ip nat inside
 ip virtual-reassembly in
!
interface GigabitEthernet0/1
 description Telstra_Fibre
 ip address 139.130.XXX.XXX 255.255.255.252
 ip nat outside
 ip virtual-reassembly in
 duplex full
 speed 1000
!
router ospf 1
 redistribute connected subnets
 redistribute static subnets
 network 10.10.10.0 0.0.0.7 area 0
 network 172.30.0.0 0.0.255.255 area 0
 default-information originate
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip nat inside source list 2 interface GigabitEthernet0/1 overload
ip nat inside source static tcp 172.22.21.118 22 139.130.XXX.XXX XXX extendable
ip nat inside source static tcp 172.22.21.118 XXXX 139.130.XXX.XXX XXX extendable
ip nat inside source static tcp 172.22.21.118 80 139.130.XXX.XXX XXX extendable
ip route 172.30.0.0 255.255.0.0 Tunnel2 track 100
ip route 172.30.0.0 255.255.0.0 Tunnel3 track 200
ip route 0.0.0.0 0.0.0.0 139.130.136.209
ip route 10.195.0.0 255.255.0.0 Tunnel5
ip route 10.197.0.0 255.255.0.0 Tunnel5
ip route 10.199.0.0 255.255.0.0 Tunnel5
ip route 10.201.0.0 255.255.0.0 Tunnel1
ip route 172.22.20.0 255.255.254.0 Tunnel4
ip route 172.29.0.0 255.255.0.0 Tunnel5
ip route 192.168.216.0 255.255.254.0 Tunnel1
!
ip sla 100
 icmp-echo 169.254.32.81 source-interface Tunnel2
 frequency 5
ip sla schedule 100 life forever start-time now
logging trap debugging
logging host 122.149.XXX.XXX
logging host 52.62.XXX.XXX
!
!
snmp-server community tres2347 RO
access-list 2 permit 192.168.212.0 0.0.0.255
access-list 2 permit 192.168.113.0 0.0.0.255
access-list 2 permit 192.168.215.0 0.0.0.255
access-list 2 permit 192.168.214.0 0.0.0.255
access-list 2 permit 192.168.213.0 0.0.0.255
access-list 2 permit 192.168.112.0 0.0.0.255
access-list 2 permit 10.10.10.0 0.0.0.7
access-list 2 permit 172.22.20.0 0.0.1.255
access-list 23 permit 122.149.XXX.XXX
access-list 23 permit 203.38.XXX.XXX
access-list 23 permit 139.130.XXX.XXX
access-list 23 permit 61.68.XXX.XXX
access-list 23 permit 10.10.10.0 0.0.0.127
access-list 23 permit 192.168.212.0 0.0.0.255
access-list 23 permit 192.168.213.0 0.0.0.255
access-list 23 permit 192.168.214.0 0.0.0.255
access-list 23 permit 172.22.20.0 0.0.1.255
access-list 60 permit 192.168.214.0 0.0.0.255
access-list 60 permit 192.168.212.0 0.0.0.255
access-list 60 permit 192.168.215.0 0.0.0.255
!
control-plane
!
!
banner exec ^C

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
XXXXXXXXX-|C|I|S|C|O|-|2|9|0|1|-|C|O|R|E|-|R|O|U|T|E|R|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

^C
banner login ^C

UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED

You must have explicit, authorized permission to access or configure this device.

Unauthorized attempts and actions to access or use this system may result in civil and/or
criminal penalties.

All activities performed on this device are logged and monitored.

^C
!
line con 0
 login local
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class 23 in
 privilege level 15
 login local
 transport input telnet ssh
line vty 5 15
 access-class 23 in
 privilege level 15
 login
 transport input telnet ssh
!
scheduler allocate 20000 1000
ntp authenticate
ntp server 130.102.2.123
!
end

[Leo Laohoo]

Kindly post the complete output to the command "sh interface <BLAH>".

[Paul Smith]

Which interface?

[Georg Pauwen]

Hello,

which tunnels are dropping ? If it are the AWS tunnels, Amazon recommends 'ip mtu 1436' and 'tcp adjust-mss 1387' on the tunnels. Configuring 'tunnel path-mtu-discovery' on the tunnel interfaces might help as well.

That said, if the tunnels are consistently dropping once a week, I am pretty sure it is due to some maintenance window on the ISP end. Are tunnels consistently dropping at a certain time ?

[Paul Smith]

Hi Georg,

Thanks, I will try those settings, although I though tcp adjust-mss was also supposed to be 40 less than ip mtu?

The dropouts are roughly once a week but never same time same day.

[Joseph W. Doherty]

. . . I though tcp adjust-mss was also supposed to be 40 less than ip mtu?

Yes, minimally.  Keep in mind IP packet headers can be larger than the minimal/usual 40 bytes.  (NB: I cannot speak on AWS tunnels.)

[paul driver]

Hello

Jus looking at you configuration are you having issues with routing tunnel 2 traffic?

Looks like you have ip sla tracking enable for network 172.30.0.0 via tunnel 2 but if his fails for what ever reason you then you want it to go over tunnel 3 but you have a track of 200 which doesn't exisit which would seem to suggest then the traffic for 172.30.0. isn't being pushed over tunnel 3 when it needs to be and goes via the default route which would then drop your ospf peering due to this incorrect re-route of traffic

Res

Paul

[Paul Smith]

Hi Paul D,

Thanks for pointing that out. That's my fault I didn't notice. It is interesting that Tunnel 2 seems to drop first. Look at these logs from the last drop which was early Sunday;

2017-04-23T00:24:11.793794+10:00 wri2519071.lnk.telstra.net 194: *Apr 23 00:25:22: %CRYPTO-6-ISAKMP_MANUAL_DELETE: IKE SA manually deleted. Do 'clear crypto sa peer 122.149.XXX.XXX' to manually clear IPSec SA's covered by this IKE SA.
2017-04-23T00:54:28.831368+10:00 wri2519071.lnk.telstra.net 195: *Apr 23 00:55:39: %TRACK-6-STATE: 100 ip sla 100 reachability Up -> Down
2017-04-23T00:55:22.127755+10:00 wri2519071.lnk.telstra.net 196: *Apr 23 00:56:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down
2017-04-23T01:20:46.592385+10:00 wri2519071.lnk.telstra.net 197: *Apr 23 01:21:56: %SYS-5-RELOAD: Reload requested by nwtech on vty0 (192.168.212.7). Reload Reason: Reload Command.
2017-04-23T01:23:38.686828+10:00 wri2519071.lnk.telstra.net 44: *Apr 23 01:25:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up
2017-04-23T01:23:41.239797+10:00 wri2519071.lnk.telstra.net 45: *Apr 23 01:25:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel5, changed state to up
2017-04-23T01:23:44.683458+10:00 wri2519071.lnk.telstra.net 46: *Apr 23 01:25:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to up
2017-04-23T01:23:44.684280+10:00 wri2519071.lnk.telstra.net 47: *Apr 23 01:25:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up
2017-04-23T01:23:44.684828+10:00 wri2519071.lnk.telstra.net 48: *Apr 23 01:25:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel4, changed state to up
2017-04-23T01:23:54.473078+10:00 wri2519071.lnk.telstra.net 49: *Apr 23 01:25:17: %TRACK-6-STATE: 100 ip sla 100 reachability Down -> Up
2017-04-23T01:23:56.536369+10:00 wri2519071.lnk.telstra.net 50: *Apr 23 01:25:19: %OSPF-5-ADJCHG: Process 1, Nbr 10.92.45.126 on GigabitEthernet0/0.125 from LOADING to FULL, Loading Done
2017-04-23T01:23:56.536369+10:00 wri2519071.lnk.telstra.net 51: *Apr 23 01:25:19: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.215.2 on GigabitEthernet0/0.125 from LOADING to FULL, Loading Done
2017-04-23T09:23:36.511907+10:00 wri2519071.lnk.telstra.net 52: *Apr 23 09:24:59: %CRYPTO-6-ISAKMP_MANUAL_DELETE: IKE SA manually deleted. Do 'clear crypto sa peer 122.149.XXX.XXX' to manually clear IPSec SA's covered by this IKE SA.

[paul driver]

Hello

Another thing is your isakmp lifetimes which if my maths is correct is set to 8 hrs and your logs do show ospf peering at 1.25 and a tear down 9:24 which would be in line with you configuration.

So are these setting comparible with the other side of the vpn and have you tried incresing the lifetimes or maybe set them to default - another would be to try disabling keepives entirely?

res

paul