10-31-2018 03:49 AM - edited 03-05-2019 11:01 AM
Hi,
We are using C2921-SECk9 router with 1GB RAM, its two or three interfaces go down every one or two months. The interfaces start working again after a reboot. What could be the cause?
System Bootstrap, Version 15.0 (1r) M15, RELEASE SOFTWARE (fc1)
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.6 (3) M, RELEASE SOFTWARE (fc1)
Last month interface Gi0/0 & 0/2 went down, this time interfaces Gi0/0 & 0/1 went down.
10-31-2018 04:19 AM
Hi Jameel,
Can you please share show interface logs prior to reboot and was other ports working at point of issue ?
10-31-2018 08:04 AM
#show ip int b Interface IP-Address OK? Method Status Protocol Embedded-Service-Engine0/0 unassigned YES NVRAM administratively down down GigabitEthernet0/0 xx.xx.xx.18 YES NVRAM down down GigabitEthernet0/1 xx.xx.xx9 YES manual down down GigabitEthernet0/1.1 unassigned YES unset down down GigabitEthernet0/2 xx.xx.xx.2 YES NVRAM up up FastEthernet0/1/0 unassigned YES unset up up FastEthernet0/1/1 unassigned YES unset up up FastEthernet0/1/2 unassigned YES unset up up FastEthernet0/1/3 unassigned YES unset up up NVI0 unassigned YES unset up up Tunnel1 xx.xx.xx.9 YES NVRAM up down Tunnel2 xx.xx.xx.9 YES NVRAM up down Tunnel3 172.16.2.2 YES NVRAM up down Vlan1 unassigned YES unset down down Vlan2 xx.xx.xx.114 YES NVRAM up up Vlan5 xx.xx.xx.142 YES NVRAM up up Vlan10 xx.xx.xx.50 YES NVRAM up up Vlan75 xx.xx.xx.122 YES NVRAM up up #conf t Enter configuration commands, one per line. End with CNTL/Z. (config)#int gi0/0 (config-if)#shut (config-if)#no shut (config-if)#^Z #show ip int b Interface IP-Address OK? Method Status Protocol Embedded-Service-Engine0/0 unassigned YES NVRAM administratively down down GigabitEthernet0/0 xx.xx.xx.18 YES NVRAM down down GigabitEthernet0/1 xx.xx.xx9 YES manual down down GigabitEthernet0/1.1 unassigned YES unset down down GigabitEthernet0/2 xx.xx.xx.2 YES NVRAM up up FastEthernet0/1/0 unassigned YES unset up up FastEthernet0/1/1 unassigned YES unset up up FastEthernet0/1/2 unassigned YES unset up up FastEthernet0/1/3 unassigned YES unset up up NVI0 unassigned YES unset up up Tunnel1 xx.xx.xx.9 YES NVRAM up down Tunnel2 xx.xx.xx.9 YES NVRAM up down Tunnel3 172.16.2.2 YES NVRAM up down Vlan1 unassigned YES unset down down Vlan2 xx.xx.xx.114 YES NVRAM up up Vlan5 xx.xx.xx.142 YES NVRAM up up Vlan10 xx.xx.xx.50 YES NVRAM up up Vlan75 xx.xx.xx.122 YES NVRAM up up #sh int gi0/0 <-- THIS INTERFACE & GI0/1 WENT DOWN GigabitEthernet0/0 is down, line protocol is down Hardware is CN Gigabit Ethernet, address is 84b8.02f2.e390 (bia 84b8.02f2.e390 ) Description: 42MBPS link Internet address is xx.xx.xx.XXX/30 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Auto Duplex, Auto Speed, media type is RJ45 output flow-control is unsupported, input flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 01:53:44, output 01:53:40, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/353 (size/max/drops/flushes); Total output drops: 135917 Queueing strategy: fifo Output queue: 0/40 (size/max) 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec 1000028587 packets input, 3309242993 bytes, 135339 no buffer Received 149347 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 3539296076 packets output, 2116284506 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 1 lost carrier, 0 no carrier, 24849 pause output 0 output buffer failures, 0 output buffers swapped out XXXXXXX#sh int gi0/2 <-- THIS INTERFACE WAS UP AND WORKING GigabitEthernet0/2 is up, line protocol is up Hardware is CN Gigabit Ethernet, address is 84b8.02f2.e392 (bia 84b8.02f2.e392 ) Description: ASA OUTSIDE Internet address is xx.xx.xx.XXX/24 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Auto Duplex, Auto Speed, media type is RJ45 output flow-control is unsupported, input flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 01:59:04, output 01:41:46, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/1806 (size/max/drops/flushes); Total output drops: 9 Queueing strategy: fifo Output queue: 0/40 (size/max) 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec 1387194064 packets input, 2665815417 bytes, 0 no buffer Received 368 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 150855 input errors, 0 CRC, 1 frame, 150853 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 3747736113 packets output, 2139195830 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 1 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out sh buffer Buffer elements: 1254 in free list 415111098 hits, 0 misses, 1242 created Public buffer pools: Small buffers, 104 bytes (total 60, permanent 50, peak 114 @ 2w3d): 31 in free list (20 min, 150 max allowed) 149871475 hits, 8751 misses, 10047 trims, 10057 created 369 failures (0 no memory) Middle buffers, 600 bytes (total 25, permanent 25, peak 74 @ 1w3d): 24 in free list (10 min, 150 max allowed) 41227352 hits, 1688 misses, 1496 trims, 1496 created 318 failures (0 no memory) Big buffers, 1536 bytes (total 48, permanent 50, peak 158 @ 7w0d): 48 in free list (5 min, 150 max allowed) 148839972 hits, 474 misses, 346 trims, 344 created 230 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 11 @ 7w0d): 10 in free list (0 min, 100 max allowed) 847 hits, 133 misses, 4 trims, 4 created 133 failures (0 no memory) Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 3w3d): 1 in free list (0 min, 10 max allowed) 8 hits, 131 misses, 710 trims, 711 created 130 failures (0 no memory) Huge buffers, 18024 bytes (total 5, permanent 0, peak 10 @ 7w0d): 5 in free list (4 min, 10 max allowed) 2821 hits, 126 misses, 1428 trims, 1433 created 115 failures (0 no memory) Interface buffer pools: CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 7w0d): 101 in free list (100 min, 200 max allowed) 0 hits, 0 misses, 706 trims, 707 created 0 failures (0 no memory) Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 7w0d): 101 in free list (100 min, 100 max allowed) 0 hits, 0 misses CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 7w0d): 101 in free list (100 min, 200 max allowed) 0 hits, 0 misses, 706 trims, 707 created 0 failures (0 no memory) Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 7w0d): 101 in free list (132 min, 132 max allowed) 1509 hits, 0 misses CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 7w0d): 26 in free list (25 min, 50 max allowed) 0 hits, 0 misses, 706 trims, 707 created 0 failures (0 no memory) IPC buffers, 4096 bytes (total 2, permanent 2): 1 in free list (1 min, 8 max allowed) 1 hits, 0 fallbacks, 0 trims, 0 created 0 failures (0 no memory) CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 7w0d): 3 in free list (2 min, 4 max allowed) 0 hits, 0 misses, 706 trims, 707 created 0 failures (0 no memory) CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 7w0d): 2 in free list (1 min, 2 max allowed) 0 hits, 0 misses, 706 trims, 707 created 0 failures (0 no memory) IPC Medium buffers, 16384 bytes (total 2, permanent 2): 2 in free list (1 min, 8 max allowed) 0 hits, 0 fallbacks, 0 trims, 0 created 0 failures (0 no memory) IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 7w0d): 17 in free list (16 min, 16 max allowed) 0 hits, 0 misses, 114235 trims, 114236 created 0 failures (0 no memory) Header pools: Header buffers, 0 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 12171333 hits in cache, 0 misses in cache Particle Clones: 1024 clones, 0 hits, 0 misses Public particle pools: F/S buffers, 1664 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 0 hits in cache, 0 misses in cache Normal buffers, 1676 bytes (total 3840, permanent 3840): 3840 in free list (128 min, 4096 max allowed) 279 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Private particle pools: HQF Particle buffers, 0 bytes (total 2000, permanent 2000): 2000 in free list (500 min, 2000 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) IDS SM buffers, 240 bytes (total 128, permanent 128): 0 in free list (0 min, 128 max allowed) 128 hits, 0 fallbacks 128 max cache size, 128 in cache 0 hits in cache, 0 misses in cache FastEthernet0/1/0 buffers, 1548 bytes (total 128, permanent 128): 0 in free list (0 min, 128 max allowed) 128 hits, 279 fallbacks 128 max cache size, 64 in cache 2789312635 hits in cache, 215 misses in cache GigabitEthernet0/0 buffers, 1664 bytes (total 1024, permanent 1024): 0 in free list (0 min, 1024 max allowed) 1024 hits, 0 fallbacks 1024 max cache size, 768 in cache 1001136832 hits in cache, 0 misses in cache GigabitEthernet0/1 buffers, 1664 bytes (total 1024, permanent 1024): 0 in free list (0 min, 1024 max allowed) 1024 hits, 0 fallbacks 1024 max cache size, 768 in cache 7359 hits in cache, 0 misses in cache GigabitEthernet0/2 buffers, 1664 bytes (total 1024, permanent 1024): 0 in free list (0 min, 1024 max allowed) 1024 hits, 0 fallbacks 1024 max cache size, 768 in cache 1387196170 hits in cache, 0 misses in cache #show logg Oct 29 19:53:58.556 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.122, prot=50, spi=0x47455420(1195725856), srcaddr=45.79.106.170, input interface=Vlan10 Oct 29 20:50:56.178 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.32, prot=50, spi=0x47455420(1195725856), srcaddr=45.79.106.170, input interface=GigabitEthernet0/0 Oct 30 02:13:22.840 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.18, prot=50, spi=0x9146DA5F(2437339743), srcaddr=58.27.215.165, input interface=GigabitEthernet0/0 Oct 30 05:28:32.533 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.144, prot=50, spi=0x30303030(808464432), srcaddr=125.212.217.214, input interface=Vlan5 Oct 30 10:25:54.431 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.145, prot=50, spi=0x47455420(1195725856), srcaddr=45.79.106.170, input interface=Vlan5 Oct 30 13:42:03.742 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.145, prot=50, spi=0x30303030(808464432), srcaddr=198.20.99.130, input interface=Vlan5 Oct 30 15:32:01.167 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.18, prot=50, spi=0x86F47665(2264168037), srcaddr=203.128.252.110, input interface=GigabitEthernet0/0 Oct 30 15:56:48.057 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.18, prot=50, spi=0x7B224248(2065842760), srcaddr=182.180.166.56, input interface=GigabitEthernet0/0 Oct 30 18:04:10.283 PST: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=xx.xx.xx.18, prot=50, spi=0x55B1163F(1437668927), srcaddr=58.27.215.165, input interface=GigabitEthernet0/0 Oct 30 20:57:33.088 PST: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down Oct 30 20:58:21.108 PST: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
10-31-2018 05:47 AM
Hello,
sounds like some sort of memory leak...
Try and upgrade to 15.7.3M3...
10-31-2018 08:08 AM
Ok, I will try upgrading the IOS. But for clarification, I wanted to ask if the memory leaks reflect in show process memory command? I checked memory right after the issue and a lot of memory was free. Here is the output.
#show process mem Processor Pool Total: 1352702428 Used: 141130184 Free: 1211572244 I/O Pool Total: 41943040 Used: 18626992 Free: 23316048 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 179834676 92309232 82031900 662 16601618 *Init* 0 0 12788 6363868 12788 41 41 *Sched* 0 0 130400404 125831484 1354648 243 241 *Dead* 0 0 0 0 591704 0 0 *MallocLite* 1 0 174919828 124940728 49986364 0 0 Chunk Manager 2 0 240 240 4264 0 0 Load Meter 3 0 29480 808 54608 0 0 LICENSE AGENT 4 0 65588 0 90852 0 0 EDDRI_MAIN 5 0 0 0 7264 0 0 RO Notify Timers 6 0 3376 240 10400 0 0 Check heaps 7 0 3417623384 3420550608 256536 49 0 Pool Manager 8 0 0 0 7264 0 0 DiscardQ Backgro 9 0 240 240 7264 0 0 Timers 10 0 0 0 4264 0 0 WATCH_AFS 11 0 0 0 7264 0 0 License Client N 12 0 0 0 7264 0 0 Image License br 13 0 27915784 27916516 7264 0 0 Licensing Auto U 14 0 183456 17316 181372 0 0 RF Slave Main Th 15 0 0 0 4264 0 0 RMI RM Notify Wa 16 0 240 240 7264 0 0 Environmental mo 17 0 0 0 7264 0 0 IPC Event Notifi 18 0 0 0 7264 0 0 IPC Dynamic Cach 19 0 0 0 7264 0 0 IPC Session Serv 20 0 0 0 7264 0 0 IPC Zone Manager 21 0 0 0 7264 0 0 IPC Periodic Tim 22 0 0 0 7264 0 0 IPC Deferred Por 23 0 0 0 7264 0 0 IPC Process leve 24 0 2104 0 9368 0 0 IPC Seat Manager 25 0 0 0 7264 0 0 IPC Check Queue 26 0 0 0 7264 0 0 IPC Seat RX Cont 27 0 0 0 7264 0 0 IPC Seat TX Cont 28 0 0 0 7264 0 0 IPC Keep Alive M 29 0 0 0 7264 0 0 IPC Loadometer 30 0 0 0 7264 0 0 IPC Session Deta 31 0 0 0 25264 0 0 Crash writer 32 0 380 0 7644 0 0 Exception contro 33 0 0 0 7128 0 0 CRYPTO IKMP IPC 34 0 0 0 7264 0 0 IFS Agent Manage 35 0 29360 12600 24024 13343527 13343527 ARP Input 36 0 177766548 177766548 7264 9333 9333 ARP Background 37 0 240 240 7264 0 0 ATM Idle Timer 38 0 0 0 7264 0 0 ATM ASYNC PROC 39 0 0 0 7264 0 0 CEF MIB API 40 0 0 0 7264 0 0 AAA_SERVER_DEADT 41 0 0 0 13264 0 0 Policy Manager 42 0 240 240 7264 0 0 DDR Timers 43 0 29024 340 36208 43 43 Entity MIB API 44 0 240 240 7264 0 0 PrstVbl 45 0 240 240 7264 0 0 Serial Backgroun 46 0 0 0 13264 0 0 License IPC stat 47 0 0 0 13264 0 0 License IPC serv PID TTY Allocated Freed Holding Getbufs Retbufs Process 48 0 240 240 7264 0 0 SMART 49 0 1688 0 8952 0 0 RF SCTPthread 50 0 1688 0 8952 0 0 CHKPT RG SCTPthr 51 0 0 0 7264 13180459 13180459 ARP Snoop 52 0 240 240 7264 0 0 GraphIt 53 0 240 240 13264 0 0 Dialer event 54 0 0 0 7264 0 0 SERIAL A'detect 55 0 240 240 13264 0 0 XML Proxy Client 56 0 0 6232 7264 0 0 crypto sw pk pro 57 0 240 240 7264 0 0 ISM-VPN main 58 0 0 0 7264 0 0 ISM-VPN Cleanup 59 0 0 0 7264 0 0 Critical Bkgnd 60 0 228776 6712 19792 2051211 2051211 Net Background 61 0 2280 2280 13264 0 0 IDB Work 62 0 240 240 13264 2939 2939 Logger 63 0 77160 10388 7264 0 0 TTY Background 65 0 1044 0 14308 0 0 CTS CORE 66 0 1044 20320 14068 0 0 SXP CORE 67 0 67132 0 74396 0 0 IF-MGR control p 68 0 27468 0 17316 36 36 IF-MGR event pro 69 0 0 0 4264 0 0 Inode Table Dest 70 0 0 0 7264 0 0 cpf_msg_holdq_pr 71 0 0 0 7264 0 0 cpf_msg_rcvq_pro 72 0 0 0 7264 0 0 cpf_process_tpQ 73 0 0 0 7264 0 0 CHKPT Test clien 74 0 0 0 7264 0 0 CHKPT Test clien 75 0 0 0 7264 0 0 CHKPT Test clien 76 0 0 0 7264 0 0 CHKPT Test clien 77 0 0 0 7264 0 0 CHKPT Test clien 78 0 0 0 7264 0 0 CHKPT Test clien 79 0 0 0 7264 0 0 CHKPT Test clien 80 0 0 0 7264 0 0 CHKPT Test clien 81 0 0 0 7264 0 0 CHKPT Test clien 82 0 0 0 7264 0 0 CHKPT Test clien 83 0 0 0 7264 0 0 CHKPT Test clien 84 0 0 0 13264 0 0 Crypto PKI-HA 85 0 0 0 7264 0 0 IKE HA Mgr 86 0 0 0 7264 0 0 IPSEC HA Mgr 87 0 240 240 7264 0 0 TDM Management 88 0 18420 10588 11084 0 0 RF Master Main T 89 0 1688 0 8952 0 0 RF Master Status 90 0 0 0 7264 8589 8589 Net Input 91 0 240 240 7264 0 0 Compute load avg 92 0 0 3692 7264 0 0 Per-minute Jobs 93 0 147728820 147600472 138612 11197661 11197661 Per-Second Jobs 94 0 0 0 7264 0 0 AggMgr Process 95 0 0 0 7264 0 0 Token Daemon 96 0 0 1838896 7264 0 0 Transport Port A 97 0 0 0 7264 0 0 HC Counter Timer 98 0 240 240 7264 0 0 SM Monitor 99 0 240 240 7264 0 0 Bryce I2C CMD Qu 100 0 240 240 13264 0 0 SM Installer 101 0 0 0 4264 0 0 dev_device_inser 102 0 0 0 4264 0 0 dev_device_remov 103 0 0 0 13264 0 0 sal_dpc_process
11-01-2018 05:43 AM
10-31-2018 03:24 PM
@Jameel Ahmed wrote:
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.6(3)M, RELEASE SOFTWARE (fc1)
I think you need to upgrade the IOS.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: