10-21-2024 08:32 AM
Hello:
I have a stack of these switches C3850 and member 1 started to reload.
Cisco IOS XE Software, Version 16.12.11
Oct 20 17:39:02.350 gmt: %SYS-2-MALLOCFAIL: Memory allocation of 2796 bytes failed from 0xAAB47E0B78, alignment 8
Pool: Processor Free: 19928 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Pool Manager", ipl= 0, pid= 10
Oct 20 17:39:11.941 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 41.227.10.57/32 into IP default routing table
Oct 20 17:39:12.001 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 41.227.9.84/32 into IP default routing table
Oct 20 17:39:32.346 gmt: %SYS-2-MALLOCFAIL: Memory allocation of 2796 bytes failed from 0xAAB47E0B78, alignment 8
Pool: Processor Free: 113176 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IOSXE-RP Punt Service Process", ipl= 0, pid= 127
Oct 20 17:39:45.830 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 197.31.3.223/32 into IP default routing table
(not enough memory to log snmp-ma trace)
(not enough memory to log snmp-ma trace)
(not enough memory to log snmp-ma trace)
Writing ThresholdFreeMeminfo to folder:file - crashinfo:threshold_lowmem_info_20241020-174038-gmt
Last reload reason : Critical software exception
10-21-2024 08:38 AM
Hi,
It appears to be a software bug. Open a ticket with TAC and give the crash file so they can analyze it and recommend a version upgrade.
HTH
10-21-2024 08:44 AM
- As the issue is mostly related to bugs usually , consider upgrading to 16.12.12 (an advisory release ) , and check if that can help ,
M.
10-21-2024 08:54 AM
Thank you for the suggestion. However, the current recommended version is 16.12.11.
10-21-2024 09:43 AM
- What is the exact model of the device (as seen from show version ) ?
M.
10-21-2024 11:25 AM
SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48T 16.12.11 CAT3K_CAA-UNIVERSALK9 INSTALL
* 2 56 WS-C3850-48T 16.12.11 CAT3K_CAA-UNIVERSALK9 INSTALL
10-21-2024 09:35 AM
This is may be bug, show version tell you what is the reason of reload.
10-21-2024 09:37 AM
Last reload reason : Critical software exception
but we need to determine the root cause of this reload
10-21-2024 10:17 AM
Are these switch connected to ISP or receiving huge routing table somehow?
10-21-2024 11:27 AM
Yes the switches are connected to ISP and might be receiving large routing tables
10-21-2024 11:44 AM - edited 10-21-2024 11:47 AM
This is crashing the switch. You need to add filter on your side or ask the ISP to put filter on their side.
Can you share the output of "show ip route summary" ?
11-06-2024 04:22 PM
In addition to the "show ip route summary" that Flavio requested, providing the "threshold_lowmem_info_20241020-174038-gmt" that was generated would be helpful as well. This is a special crashinfo file that gets generated in a low memory scenario, and it will help us get a better understanding of what process is holding a significant amount of memory, which led to the memory allocation (MALLOC) failures (MALLOCFAIL) seen in the snippet of logs you provided.
If the switches are receiving more routes than they can handle, that would explain the memory exhaustion and subsequent crash.
The following are also good commands to collect on a live box to check the memory status at the current moment in time:
show clock
show version
show platform software status control-processor brief
show processes memory sorted
! Checks IOSd memory utilization
show processes memory platform sorted
! Checks kernel memory utilization
The threshold_lowmem_info file along with any crash-related files (crashinfo, core.gz, system-reports, etc.) are your best bet for getting confirmation.
11-06-2024 05:17 PM
If this switch/stack was on 16.12.11, what was the uptime before the crash?
11-07-2024 06:02 AM
You can confirm the previous uptime of the switch using the following command:
show logging onboard switch <active|standby|#> uptime detail
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide