I have a customer with a Cisco 4331 router. The symptoms on the surface is that this location is loosing the internet repeatedly, over and over again. If first started with about a 5-10 minute outage once or twice an hour. Now they go down frequently over and over again, so they are only up about 5 times an hour for a few minutes each. I monitor their equipment with PRTG that has sensors reporting ping and bandwidth. This is how I know how much they go down, along with reports from the customer on site.
I've contacted and supplied information to our ISP, which is two parts. One fiber supplier for layer 2, they've been looking at it with their engineers for several days now and haven't found any issues with their lines or switches on their MPLS network. The other partner that supplies their bandwidth, I had them check for attacks or anything that they could see. They see nothing. They put them into DDOS mitigation in case they could scrub out some of the bad traffic, but this hasn't changed how much the site looses the internet in a given hour.
Its possible either the Cisco router is getting hit, or the next hop, their Cisco ASA firewall is getting hit an attack of some kind. This is my best guess. However, I normally monitor the link from the router to the firewall, not to the ISP. So, if that link goes down then I'm not always so sure if its the firewall or our router. So, I added the link to the ISP equipment to see whats going down and when. The problem I have is that not only is the link to the firewall going down, so is the link to the ISP. And if the ISP isn't having an issue then is the router the issue? Because if he's getting attacked at the Cisco firewall with port scans or something of that nature, then why would my router be affected. They are not saturating their circuit. Because both of my interfaces go down, I'm start to question the status/health of my router.
Which brought me to two other possibilities. One, the router needs to be patched. I went to download the software for it and I cannot according the portal. This is odd since support confirmed I have Smartnet on the router. Two, its malfunctioning in some way. The only thing is I don't know all of the diagnostics commands to look for issues. The logs have nothing but OSPF messages right now. If I say show log. The up time on the router is 100%, it hasn't rebooted in months. So, its not cycling power or crashing in that way. If the interfaces are resetting, I don't know how to tell other than show interface commands.
On the interface facing the ISP I see input errors, overruns and pause outputs. I normally don't see this, but maybe its nothing.
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
30 second input rate 16507000 bits/sec, 2149 packets/sec
30 second output rate 9840000 bits/sec, 1958 packets/sec
27211280955 packets input, 29649530972733 bytes, 0 no buffer
Received 374658314 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
78 input errors, 0 CRC, 0 frame, 78 overrun, 0 ignored
0 watchdog, 935278023 multicast, 0 pause input
20973626015 packets output, 13076748351934 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 1032 pause output
0 output buffer failures, 0 output buffers swapped out
Are their any obvious things I may be missing in looking into this. At this point I'm suspecting the router is doing something wrong. If its not the router, then maybe its just an attack I cannot do anything about.
Thanks,
Mike
