We have an ISR4321/k9 installed at one of our colo's and it's been working just fine for the better part of two months. recently we started having issues with traffic suddenly not getting through. I can still telnet into the router from an external IP, and the server logs show that the servers on the inside never lost internet connectivity (though I'm not sure if we have a great litmus test in place for this), but nothing outside can connect to the servers.
we're using nat to translate a set of public IPs and ports to the local IPs, so I figured this was part of the problem. google told me to check the logs and I found this:
*Oct 29 18:58:21.459: %IOSXE-4-PLATFORM: SIP1: cpp_cp: QFP:0.0 Thread:000 TS:000 00015964780136856 %NAT-4-DEFAULT_MAX_ENTRIES: default maximum entries value 131072 exceeded; frame dropped
Google also told me to run the following commands to resolve the issue:
# ip nat translation max-entries 247483647
# clear ip nat translation *
Those seemed to do the trick and we opted to subscribe to a domain checking service to monitor the site for outages. Things ran for almost a week without incident!
...this morning, I woke up with my inbox blown to pieces and the site down again.
Logged in and ran sh log, but I didn't see anything new... I didn't even see myself logging in! I should have cleared the log, waited a minute and checked again, but I went ahead and ran clear ip nat translation *, and of course the site immediately came back online, so I'm pretty sure the NAT translation table filled up again.
I'm fairly familiar with cisco switches, but routing is a new beast for me so I'm pretty out of my depth on this one.
What do I need to do to stop this from happening?
In case it's useful, here's my running config:
colo-net1#sh run
Building configuration...
Current configuration : 4317 bytes
!
! Last configuration change at 16:21:57 UTC Fri Nov 1 2019
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname colo-net1
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
enable secret 5 [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
enable password [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
!
no aaa new-model
!
transport-map type persistent webui https-webui
secure-server
!
transport-map type persistent webui http-webui
server
!
transport-map type persistent webui http-https-webui
server
secure-server
!
!
!
!
!
!
!
!
!
!
!
!
ip name-server 208.67.222.222
!
ip dhcp pool 1
utilization mark high 80 log
utilization mark low 70 log
network 192.168.60.0 255.255.255.0
domain-name int.pos.ac
default-router 192.168.60.1
dns-server 208.67.222.222 8.8.8.8
lease 30
class CLASSC
address range 192.168.60.240 192.168.60.250
!
!
ip dhcp class CLASSC
!
!
!
!
!
!
!
!
!
subscriber templating
multilink bundle-name authenticated
!
!
!
crypto pki trustpoint TP-self-signed-1403732793
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-1403732793
revocation-check none
rsakeypair TP-self-signed-1403732793
!
!
crypto pki certificate chain TP-self-signed-1403732793
license udi pid ISR4331/K9 sn [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
!
spanning-tree extend system-id
!
username vadmin privilege 15 secret 5 [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
!
redundancy
mode none
!
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface GigabitEthernet0/0/0
description WAN000
ip address XXX.XXX.XXX.37 255.255.255.240 secondary
ip address XXX.XXX.XXX.38 255.255.255.240 secondary
ip address XXX.XXX.XXX.39 255.255.255.240 secondary
ip address XXX.XXX.XXX.40 255.255.255.240 secondary
ip address XXX.XXX.XXX.41 255.255.255.240 secondary
ip address XXX.XXX.XXX.42 255.255.255.240 secondary
ip address XXX.XXX.XXX.43 255.255.255.240 secondary
ip address XXX.XXX.XXX.44 255.255.255.240 secondary
ip address XXX.XXX.XXX.45 255.255.255.240 secondary
ip address XXX.XXX.XXX.46 255.255.255.240 secondary
ip address XXX.XXX.XXX.36 255.255.255.240
ip nat outside
negotiation auto
!
interface GigabitEthernet0/0/1
ip address 192.168.61.1 255.255.255.0
negotiation auto
!
interface GigabitEthernet0/0/2
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet0/1/0
!
interface GigabitEthernet0/1/1
!
interface GigabitEthernet0/1/2
!
interface GigabitEthernet0/1/3
!
interface GigabitEthernet0/1/4
!
interface GigabitEthernet0/1/5
!
interface GigabitEthernet0/1/6
!
interface GigabitEthernet0/1/7
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address 192.168.60.3 255.255.255.0
negotiation auto
!
interface Vlan1
ip address 192.168.60.1 255.255.255.0
ip nat inside
!
ip nat translation timeout 86400
ip nat translation max-entries 247483647
ip nat pool NAT1 XXX.XXX.XXX.36 XXX.XXX.XXX.36 netmask 255.255.255.240
ip nat inside source static 192.168.60.11 XXX.XXX.XXX.39 extendable
ip nat inside source static 192.168.60.21 XXX.XXX.XXX.40 extendable
ip nat inside source static 192.168.60.119 XXX.XXX.XXX.41 extendable
ip nat inside source static 192.168.60.120 XXX.XXX.XXX.42 extendable
ip nat inside source static 192.168.60.121 XXX.XXX.XXX.43 extendable
ip nat inside source static 192.168.60.122 XXX.XXX.XXX.44 extendable
ip nat inside source static 192.168.60.123 XXX.XXX.XXX.45 extendable
ip nat inside source static 192.168.60.124 XXX.XXX.XXX.46 extendable
ip nat inside source static udp 192.168.60.11 1195 interface GigabitEthernet0/0/0 1195
ip nat inside source static udp 192.168.60.5 1192 interface GigabitEthernet0/0/0 1192
ip nat inside source static udp 192.168.60.5 1194 interface GigabitEthernet0/0/0 1194
ip nat inside source static tcp 192.168.60.11 10000 interface GigabitEthernet0/0/0 10000
ip nat inside source list 1 interface GigabitEthernet0/0/0 overload
ip forward-protocol nd
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 XXX.XXX.XXX.33
!
!
access-list 1 permit 192.168.60.0 0.0.0.255
!
snmp-server community public RO
!
!
control-plane
!
!
line con 0
password [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
login
stopbits 1
line aux 0
stopbits 1
line vty 0 4
password [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
login
!
transport type persistent webui input https-webui
!
!
end
