topic Re: C3850 Reload unexpended in Switching

C3850 Reload unexpended

RBHRBHRBH — Mon, 21 Oct 2024 15:32:56 GMT

Hello:

I have a stack of these switches C3850 and member 1 started to reload.

Cisco IOS XE Software, Version 16.12.11

Oct 20 17:39:02.350 gmt: %SYS-2-MALLOCFAIL: Memory allocation of 2796 bytes failed from 0xAAB47E0B78, alignment 8
Pool: Processor Free: 19928 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Pool Manager", ipl= 0, pid= 10
Oct 20 17:39:11.941 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 41.227.10.57/32 into IP default routing table
Oct 20 17:39:12.001 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 41.227.9.84/32 into IP default routing table
Oct 20 17:39:32.346 gmt: %SYS-2-MALLOCFAIL: Memory allocation of 2796 bytes failed from 0xAAB47E0B78, alignment 8
Pool: Processor Free: 113176 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IOSXE-RP Punt Service Process", ipl= 0, pid= 127
Oct 20 17:39:45.830 gmt: %IPRT-3-ROUTEINSERTERROR: Error inserting routing entry 197.31.3.223/32 into IP default routing table
(not enough memory to log snmp-ma trace)
(not enough memory to log snmp-ma trace)
(not enough memory to log snmp-ma trace)

Writing ThresholdFreeMeminfo to folder:file - crashinfo:threshold_lowmem_info_20241020-174038-gmt

Last reload reason : Critical software exception

Re: C3850 Reload unexpended

Reza Sharifi — Mon, 21 Oct 2024 15:38:18 GMT

Hi,

It appears to be a software bug. Open a ticket with TAC and give the crash file so they can analyze it and recommend a version upgrade.

HTH

Re: C3850 Reload unexpended

Mark Elsen — Mon, 21 Oct 2024 15:44:49 GMT

- As the issue is mostly related to bugs usually , consider upgrading to 16.12.12 (an advisory release ) , and check if that can help ,

Re: C3850 Reload unexpended

RBHRBHRBH — Mon, 21 Oct 2024 15:54:51 GMT

Thank you for the suggestion. However, the current recommended version is 16.12.11.

Re: C3850 Reload unexpended

balaji.bandi — Mon, 21 Oct 2024 16:35:17 GMT

This is may be bug, show version tell you what is the reason of reload.

Re: C3850 Reload unexpended

RBHRBHRBH — Mon, 21 Oct 2024 16:37:02 GMT

Last reload reason : Critical software exception
but we need to determine the root cause of this reload

Re: C3850 Reload unexpended

Mark Elsen — Mon, 21 Oct 2024 16:43:47 GMT

- What is the exact model of the device (as seen from show version ) ?

Re: C3850 Reload unexpended

Flavio Miranda — Mon, 21 Oct 2024 17:17:41 GMT

@RBHRBHRBH

Are these switch connected to ISP or receiving huge routing table somehow?

Re: C3850 Reload unexpended

RBHRBHRBH — Mon, 21 Oct 2024 18:25:54 GMT

SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48T 16.12.11 CAT3K_CAA-UNIVERSALK9 INSTALL
* 2 56 WS-C3850-48T 16.12.11 CAT3K_CAA-UNIVERSALK9 INSTALL

Re: C3850 Reload unexpended

RBHRBHRBH — Mon, 21 Oct 2024 18:27:11 GMT

Yes the switches are connected to ISP and might be receiving large routing tables

Re: C3850 Reload unexpended

Flavio Miranda — Mon, 21 Oct 2024 18:47:31 GMT

This is crashing the switch. You need to add filter on your side or ask the ISP to put filter on their side.

Can you share the output of "show ip route summary" ?

Re: C3850 Reload unexpended

dmonazah — Thu, 07 Nov 2024 00:22:30 GMT

In addition to the "show ip route summary" that Flavio requested, providing the "threshold_lowmem_info_20241020-174038-gmt" that was generated would be helpful as well. This is a special crashinfo file that gets generated in a low memory scenario, and it will help us get a better understanding of what process is holding a significant amount of memory, which led to the memory allocation (MALLOC) failures (MALLOCFAIL) seen in the snippet of logs you provided.

If the switches are receiving more routes than they can handle, that would explain the memory exhaustion and subsequent crash.

The following are also good commands to collect on a live box to check the memory status at the current moment in time:

show clock
show version
show platform software status control-processor brief
show processes memory sorted
    ! Checks IOSd memory utilization
show processes memory platform sorted
    ! Checks kernel memory utilization

The threshold_lowmem_info file along with any crash-related files (crashinfo, core.gz, system-reports, etc.) are your best bet for getting confirmation.

Re: C3850 Reload unexpended

Leo Laohoo — Thu, 07 Nov 2024 01:17:03 GMT

If this switch/stack was on 16.12.11, what was the uptime before the crash?

Re: C3850 Reload unexpended

dmonazah — Thu, 07 Nov 2024 14:02:22 GMT

You can confirm the previous uptime of the switch using the following command:

show logging onboard switch <active|standby|#> uptime detail