C7301-K91P-M, Version 12.2(18)S11 - shuts down all interfaces/peers

[georgepatterson]

Hello,

I have a Cisco 7301 running release IOS (tm) 7301 Software (C7301-K91P-M), Version 12.2(18)S11, which has some updates available. The device has been stable for over a year and a half, but our traffic volume has been growing. We have 3 transit providers that we speak BGP with, lately we've started seeing a strange issue where all neighbors will shutdown at the same time and will not restablish their session. We're working on getting an out-of-band line at the site, but for now we've just had remote hands reboot the device. Soon as it comes up, we're able to access the device on all of the direct IPs for each provider, and our range starts to be announced again. If we have the tech plug in a console cable to the router, he's able to get a prompt with updates saying all of the interfaces have changed to down.

Has anyone seen anything like this or have thoughts on what could be happening? There's some examples of what's logged below, we have a number of local peers that we talk to, those also all change to down with the same messages.

Dec 19 14:42:22 x.x.x.1 1772: Dec 19 14:42:21.380 UTC: %DUAL-5-NBRCHANGE: IP-EIGRP(0) xxxxx: Neighbor b.b.b.33 (Tunnel1004) is down: holding time expired

Dec 19 14:42:24 x.x.x.1 1773: Dec 19 14:42:23.556 UTC: %DUAL-5-NBRCHANGE: IP-EIGRP(0) xxxxx: Neighbor b.b.b.9 (Tunnel4) is down: holding time expired

Dec 19 14:42:26 x.x.x.1 1774: Dec 19 14:42:25.020 UTC: %DUAL-5-NBRCHANGE: IP-EIGRP(0) xxxxx: Neighbor b.b.b.73 (Tunnel504) is down: holding time expired

Dec 19 14:43:36 x.x.x.1 1775: Dec 19 14:43:36.444 UTC: %BGP-5-ADJCHANGE: neighbor c.c.c.229 Down BGP Notification sent

Dec 19 14:43:37 x.x.x.1 1776: Dec 19 14:43:36.444 UTC: %BGP-3-NOTIFICATION: sent to neighbor c.c.c.229 4/0 (hold time expired) 0 bytes

Dec 19 14:45:09 x.x.x.1 1777: Dec 19 14:45:09.364 UTC: %BGP-5-ADJCHANGE: neighbor d.d.d.113 Down BGP Notification sent

Dec 19 14:45:10 x.x.x.1 1778: Dec 19 14:45:09.364 UTC: %BGP-3-NOTIFICATION: sent to neighbor e.e.e.113 4/0 (hold time expired) 0 bytes

[shrikar.dange]

hi,

Can you check the CPU,buffer and memory utilisation of the router?

Regards,

shri.

[georgepatterson]

See attached, was too big to paste in to comment/post.

The CPU does spike a bit high at times. We're running about 100 megs through it, but a large amount of that is really small UDP packets.

Here's the interface for the switch:

GigabitEthernet0/0 is up, line protocol is up

Hardware is BCM1250 Internal MAC, address is

Description: trunk to sw1

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 25/255, rxload 25/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is unknown media type

output flow-control is off, input flow-control is off

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/30603/5002886 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 98190000 bits/sec, 67888 packets/sec

5 minute output rate 98094000 bits/sec, 67458 packets/sec

1349300051 packets input, 709195946 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicast)

0 runts, 0 giants, 30498 throttles

0 input errors, 0 CRC, 0 frame, 18 overrun, 0 ignored

0 watchdog, 5561348 multicast, 0 pause input

1034612906 packets output, 3593538737 bytes, 0 underruns

0 output errors, 0 collisions, 4 interface resets

0 babbles, 0 late collision, 0 deferred

5 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out