cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1165
Views
0
Helpful
13
Replies

C3850 Reload unexpended

RBHRBHRBH
Level 1
Level 1

Hello:

I have a stack of these switches C3850  and member 1 started to reload.

Cisco IOS XE Software, Version 16.12.11

Oct 20 17:39:02.350 gmt: %SYS-2-MALLOCFAIL: Memory allocation of 2796 bytes failed from 0xAAB47E0B78, alignment 8
Pool: Processor Free: 19928 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Pool Manager", ipl= 0, pid= 10
Oct 20 17:39:11.941 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 41.227.10.57/32 into IP default routing table
Oct 20 17:39:12.001 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 41.227.9.84/32 into IP default routing table
Oct 20 17:39:32.346 gmt: %SYS-2-MALLOCFAIL: Memory allocation of 2796 bytes failed from 0xAAB47E0B78, alignment 8
Pool: Processor Free: 113176 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IOSXE-RP Punt Service Process", ipl= 0, pid= 127
Oct 20 17:39:45.830 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 197.31.3.223/32 into IP default routing table
(not enough memory to log snmp-ma trace)
(not enough memory to log snmp-ma trace)
(not enough memory to log snmp-ma trace)

 

Writing ThresholdFreeMeminfo to folder:file - crashinfo:threshold_lowmem_info_20241020-174038-gmt

 

Last reload reason : Critical software exception

 

13 Replies 13

Reza Sharifi
Hall of Fame
Hall of Fame

Hi,

It appears to be a software bug. Open a ticket with TAC and give the crash file so they can analyze it and recommend a version upgrade.

HTH

marce1000
Hall of Fame
Hall of Fame

 

 - As the issue is mostly related to bugs usually  , consider upgrading to 16.12.12 (an advisory release ) , and check if that can help ,

 M.



-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
    When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Thank you for the suggestion. However, the current recommended version is 16.12.11. 

 

 - What is the exact model of the device (as seen from show version ) ?

 M.



-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
    When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48T 16.12.11 CAT3K_CAA-UNIVERSALK9 INSTALL
* 2 56 WS-C3850-48T 16.12.11 CAT3K_CAA-UNIVERSALK9 INSTALL

balaji.bandi
Hall of Fame
Hall of Fame

This is may be bug, show version tell you what is the reason of reload.

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Last reload reason : Critical software exception 
but we need to determine the root cause of this reload

@RBHRBHRBH 

 Are these switch connected to ISP or receiving huge routing table somehow?

RBHRBHRBH
Level 1
Level 1

Yes the switches are connected to ISP and might be receiving large routing tables

This is crashing the switch. You need to add filter on your side or ask the ISP to put filter on their side. 

Can you share the output of "show ip route summary" ?

dmonazah
Cisco Employee
Cisco Employee

In addition to the "show ip route summary" that Flavio requested, providing the "threshold_lowmem_info_20241020-174038-gmt" that was generated would be helpful as well. This is a special crashinfo file that gets generated in a low memory scenario, and it will help us get a better understanding of what process is holding a significant amount of memory, which led to the memory allocation (MALLOC) failures (MALLOCFAIL) seen in the snippet of logs you provided.

If the switches are receiving more routes than they can handle, that would explain the memory exhaustion and subsequent crash.

The following are also good commands to collect on a live box to check the memory status at the current moment in time:

show clock
show version
show platform software status control-processor brief
show processes memory sorted
! Checks IOSd memory utilization
show processes memory platform sorted
! Checks kernel memory utilization

The threshold_lowmem_info file along with any crash-related files (crashinfo, core.gz, system-reports, etc.) are your best bet for getting confirmation. 

Leo Laohoo
Hall of Fame
Hall of Fame

If this switch/stack was on 16.12.11, what was the uptime before the crash?

dmonazah
Cisco Employee
Cisco Employee

You can confirm the previous uptime of the switch using the following command: 

show logging onboard switch <active|standby|#> uptime detail