Cisco2951 input hangs. I can SSH to outside and reload.

[chris@routerguy.com]

So, I have this CISCO2951/K9.  On the "inside" interface, it locks up and will not accept traffic.  I can try to clear the interface, shutdown and no shutdown, I can reset and try anything.

The worst thing of this situation is, you never know when this failure will happen randomly.  When you are least expecting it.  I can go for days, weeks or even a couple months with no failures.  Then BOOM!

CPU load is OK.  But no traffic will pass from Gig 0/1 (inside) to the rest of the world or the inner workings of the router.

I can connect to the outside (gig 0/0) interface and issue any command I wish.  The CPU load is fine.  The interface is showing no traffic at all and is showing up.  I tried researching wedged interface soltutions.

This unit is running Version 15.7(3)M4

Any hints or suggestions are welcome.

[Richard Burts]

The next time that this happens would you post the output of these commands

show ip interface brief

show ip route

show arp

the first several pages of show log

[chris@routerguy.com]

And of course, Richard, I did not see your request for information after it happened this morning.  But I did capture the show interface.

Notice the input queue below:

MRW-2951#sh int gig 0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is PQ3_TSEC, address is e4d3.f1a7.cb51 (bia e4d3.f1a7.cb51)
Description: Lab Edge
Internet address is 192.168.120.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:38, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 76/75/10144/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 157000 bits/sec, 15 packets/sec
10640279 packets input, 6816819519 bytes, 0 no buffer
Received 334774 broadcasts (166513 IP multicasts)
0 runts, 0 giants, 7630 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 249230 multicast, 0 pause input
11478491 packets output, 10036693913 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
5655 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
MRW-2951#

[Richard Burts]

The next time it happens please post all of the items that I requested. The output that you posted does have several interesting things. Looking at the input queue the current size is 76 and the max size is 75. So it looks like your peer on that interface is sending more quickly that you can receive and process. This is confirmed by seeing 10144 input drops. It is interesting to see that you have received 7630 throttles, which suggests some capacity mismatch on their side also. Do you know how long this interface has been up (how long have the drops and throttles been accumulating)? What can you tell us about this peer and this connection?

[chris@routerguy.com]

Hello Richard,

This is the LAN (inside) interface of my 2951 firewall. Outside is a 30mbit link (all I can get where I live)

The interface and this router has been operation in this environment for about 5 years. The hangs happen about once every couple of months, however as of this posting, I had two in as many days. Very odd.

It is tied to a Cisco 3750 switch and I have a PC, a small “server” I use for storage (think NAS), wireless stuff and iPads, iPhones connected to the wireless. Pretty simple network. Oh, and a permanent IKEv1 VPN to a friend.

Thanks,

Chris

[marce1000]

  - Configure and or use a syslog server on the device then examine logs send to it when this problem happens, also use https://software.cisco.com/download/home/282774230/type/280805680/release/15.7.3M8  (latest advisory) , 

 M.

[chris@routerguy.com]

Thank you, M for the idea.  I'll have a syslog server.  Just need to research what debug i should enable.  I do not have a smartnet on this so I am not eligible to download this latest version.  I'll have to work around with what I have.

[marce1000]

                 >... I'll have to work around with what I have.
  - If the problem is due to a native bug then that is a showstopper, ...

 M.

[chris@routerguy.com]

Hello M,

This is why I was trying to research if this is a known hardware issue on Cisco's Bug Tool.  However, it looks like I have been locked out of this section as well.  

So no researching any further on this topic on Cisco.com for me.

Thanks for your time.  I hope I can find a solution.

[marce1000]

                                     >....  I hope I can find a solution.
  - I beg my pardon so to speak , but you as good as won't , because in 99% of cases these are software bugs and then going to latest advisory release is always recommended , of course one could argument  the absence of  a smartnet contract (e.g.) , but then it all depends on how important all of this (device) becomes for your business because then management could assert this value and take the contract (e.g.)  , 

 M.

[chris@routerguy.com]

Hi!

I agree getting smartnet would be great, but I believe the 2951 is now EOS… So not sure if I can even get Smartnet at this juncture…

Chris

[marce1000]

                                         >...but I believe the 2951 is now EOS…
  - For EOS info's on your device(s) , enter the serial number of the 2951 in this app https://cway.cisco.com/sncheck/

 M.

[chris@routerguy.com]

Thank you for that link. I checked. Alas, it has passed time:

End of Support 2022-12-31<>

[marce1000]

                      >...End of Support 2022-12-31<>
  - In the same context and from a business viewpoint, you may want to replace the device with supported model include support :
                                     https://www.cisco.com/c/en/us/products/routers/router-selector.html
                                     https://www.cisco.com/c/en/us/solutions/small-business/networking/routers.html

 M.

[MHM Cisco World]

interface is UP but not accept any traffic 
check first IP you assign to interface is conflict with other interface