2900 router hangs and users cannot connect to the internet but can ping internet from the router

tiroyaone · ‎08-13-2019

we have a 2900 router that hangs every morning and cannot connect users to the internet even though we can ping the internet from the router. it only start working after a command reload: Kindly assist.

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Fri 05-Jun-15 13:24 by prod_rel_team

ROM: System Bootstrap, Version 15.0(1r)M16, RELEASE SOFTWARE (fc1)

GATEWAY uptime is 6 minutes

System returned to ROM by reload at 08:07:38 GMT Wed Aug 14 2019
System restarted at 08:09:38 GMT Wed Aug 14 2019
System image file is "flash0:c2900-universalk9-mz.SPA.154-3.M3.bin"
Last reload type: Normal Reload
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco CISCO2911/K9 (revision 1.0) with 483328K/40960K bytes of memory.
Processor board ID FGL200710QZ
5 Gigabit Ethernet interfaces
1 terminal line
DRAM configuration is 64 bits wide with parity enabled.
255K bytes of non-volatile configuration memory.
255488K bytes of ATA System CompactFlash 0 (Read/Write)

License Info:

License UDI:

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*1 CISCO2911/K9 FGL200710QZ

Technology Package License Information for Module:'c2900'

------------------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------
ipbase ipbasek9 Permanent ipbasek9
security None None None
uc None None None
data None None None
NtwkEss None None None
CollabPro None None None

Configuration register is 0x2102

GATEWAY#sh log
Syslog logging: enabled (0 messages dropped, 3 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 43 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 43 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level informational, 46 message lines logged
Logging Source-Interface: VRF Name:

Log Buffer (4096 bytes):

Aug 13 17:41:09.079: %DHCPD-4-PING_CONFLICT: DHCP address conflict: server pinged 10.26.0.48.
GATEWAY# sh int gi0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is 00c8.8b9f.8f31 (bia 00c8.8b9f.8f31)
Description: *** LINK-TO-RADIO-SERVICES ****
MTU 9500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:14:08
Input queue: 0/1000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
5 minute input rate 48000 bits/sec, 48 packets/sec
5 minute output rate 11000 bits/sec, 13 packets/sec
89805146 packets input, 2872904286 bytes, 0 no buffer
Received 479378 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 30577 multicast, 0 pause input
153013202 packets output, 2189897664 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
40894 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GATEWAY# sh int gi0/0/0
GigabitEthernet0/0/0 is up, line protocol is up
Hardware is EHWIC-1GE-SFP-CU, address is 00c8.8b9f.8f38 (bia 00c8.8b9f.8f38)
Description: *** LINK_TO_ADMIN_POE_SWITCHES ****
MTU 9500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is LX
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:14:13
Input queue: 0/1000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
5 minute input rate 35000 bits/sec, 36 packets/sec
5 minute output rate 58000 bits/sec, 16 packets/sec
6516711 packets input, 2752015634 bytes, 0 no buffer
Received 20430 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
12736550 packets output, 3819923120 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
2417 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GATEWAY# sh int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Hardware is EHWIC-1GE-SFP-CU, address is 00c8.8b9f.8f40 (bia 00c8.8b9f.8f40)
Description: *** FIBRE UPLINK: -> IPCORE ***
MTU 9500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:14:17
Input queue: 0/1000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
5 minute input rate 67000 bits/sec, 38 packets/sec
5 minute output rate 33000 bits/sec, 36 packets/sec
165734486 packets input, 1386676346 bytes, 0 no buffer
Received 2069 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
95056993 packets output, 788179852 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
2069 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

GATEWAY#sh ip traffic
IP statistics:
Rcvd: 10778963 total, 2539015 local destination
36 format errors, 2 checksum errors, 308 bad hop count
2 unknown protocol, 37 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment
Bcast: 251095 received, 6725 sent
Mcast: 0 received, 0 sent
Sent: 2382644 generated, 584997283 forwarded
Drop: 6510 encapsulation failed, 0 unresolved, 0 no adjacency
4630475 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 315 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
1014 physical broadcast
Reinj: 0 in input feature path, 2391 in output feature path

ICMP statistics:
Rcvd: 1 format errors, 27 checksum errors, 0 redirects, 8183 unreachable
721170 echo, 25 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
1258 time exceeded, 0 info replies
Sent: 0 redirects, 587933 unreachable, 6601 echo, 721170 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 308 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 887080 total, 14094 checksum errors, 811477 no port 0 finput
Sent: 56107 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

TCP statistics:
Rcvd: 921798 total, 1546 checksum errors, 15630 no port
Sent: 1014537 total

EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 1340777 requests, 125 replies, 0 reverse, 0 other
Sent: 22595 requests, 606919 replies (242 proxy), 0 reverse
Drop due to input queue full: 170

after reload command:

GATEWAY#sh processes cpu history

GATEWAY 08:30:12 AM Wednesday Aug 14 2019 GMT

11111111111111111111111111111 111111111111111 111111
666666666888888888811111111118888800000444441111199999111114
100
90
80
70
60
50
40
30
20 *******************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

1221221222121122126
7207037783915172897
100
90
80
70 *
60 *
50 *
40 *
30 ** * **
20 ****##**###** *#**#
10 ###################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)

Mark Elsen · ‎08-14-2019

- Activate syslogging to a syslog server and snmp trap forwarding to a trap receiver, watch for alerts on this platform at a regular basis. Particularly when the router hangs. Also check current software version, if applicable , use a more recent release and try again.

M.

-- Let everything happen to you
   Beauty and terror
      Just keep going
     No feeling is final
Reiner Maria Rilke (1899)

tiroyaone · ‎08-14-2019

thank you for the response,

There were no logs, see below the logs before we reloaded the router: