Switches hang/crashes and needs restarting after firmware upgrade

[Ken_10]

Hi,

I would like to check what could be causing a stacked switch to crash/hang and needs restarting frequently. This started after we upgraded our stack of WS-C2960X-24TS-L from 152-7.E4 to 15.2(7)E9, and this still occurs after I've replaced the switches with 2 replacement switches of the same model and upgraded the firmware to 15.2(7)E10.

I've attached part of the crash files output below:

Apr 28 07:52:52.279: %SYS-2-MALLOCFAIL: Memory allocation of 45808 bytes failed from 0x38EA3A4, alignment 0 (Redacted)
Pool: Processor Free: 434248 Cause: Memory fragmentation (Redacted)
Alternate Pool: None Free: 0 Cause: No Alternate pool (Redacted)
-Process= "HQM Stack Process", ipl= 0, pid= 195 (Redacted)
-Traceback= 7E951Cz 2D57120z 2D5FDF8z 3DD60F4z 38EA3A8z 38C4EA8z 3B9327Cz 3B8EF6Cz (Redacted)**Panic**(HRPC) System unable to alloc memory (sz = 22960) for RPC 14:37 (Redacted)
(Redacted)
**Panic**(HRPC) System unable to alloc memory (sz = 22960) for RPC 14:37 (Redacted - Switch Name)
(Redacted)
Apr 28 07:53:03.597: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for hrpc frags. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1 (Redacted)
-Traceback= 7E951Cz 2D542B0z 2D5451Cz 3B9327Cz 3B8EF6Cz (Redacted)
Apr 28 07:53:13.940: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for hrpc frags. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1 (Redacted)
-Traceback= 7E951Cz 2D542B0z 2D5451Cz 3B9327Cz 3B8EF6Cz (Redacted)
Apr 28 07:53:03.597: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for hrpc frags. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1 (Redacted)
-Traceback= 7E951Cz 2D542B0z 2D5451Cz 3B9327Cz 3B8EF6Cz (Redacted)
Apr 28 07:53:13.940: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for hrpc frags. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1 (Redacted)
-Traceback= 7E951Cz 2D542B0z 2D5451Cz 3B9327Cz 3B8EF6Cz (MY_TMIM_Interconnect-2)**Panic**(HRPC) System unable to alloc memory (sz = 22960) for RPC 14:37 (Redacted)
(Redacted)

The same 15.2(7)E9 firmware was used without any issue on other switches. I am checking here for any pottential causes as these switches.

[balaji.bandi]

Not sure what causing the issue you have some options as below :

1. If this major impacting environment, role back to old version, you get some time to investigate.

2. or contact TAC immediatly to assists on this.

Other options ;

%SYS-2-MALLOCFAIL: Memory allocation of 45808 bytes failed from 0x38EA3A4, alignment 0 (Redacted)
Pool: Processor Free: 434248 Cause: Memory fragmentation (Redacted)

These messages indicate that a process was unable to find a large enough contiguous block of memory. In this example, the IP input process attempted to get 1028 bytes from the Processor pool of memory.

check the below thread :

https://community.cisco.com/t5/routing/sys-2-mallocfail-error/td-p/1385432

This could be  a bug also :

https://bst.cisco.com/bugsearch/bug/CSCuo46190?rfs=qvlogin

[Ken_10]

Unfortunately the device is not under any contract and I couldn't raise a TAC case for it, which is why I am looking for a solution here.

[Mark Elsen]

                       >...Cause: Memory fragmentation (Redacted)
  - Could you try a full power cycle (cold start) from the involved devices and check if that can help ,

 M.

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
dir
sh log on uptime det

[Ken_10]

I have one of the requested logs currently, as the switch is deployed at a customer site.

MY_TMIM_Interconnect#dir
Directory of flash:/

2 -rwx 26786816 Jul 13 2000 12:41:39 +08:00 c2960x-universalk9-mz.152-7.E10.bin
647 drwx 512 Jul 13 2000 12:25:44 +08:00 pnp-info
3 drwx 512 Jul 13 2000 12:26:16 +08:00 pnp-tech
4 -rwx 1936 Jul 13 2000 12:28:25 +08:00 vlan.dat
5 -rwx 156 Jul 9 2018 05:30:08 +08:00 express_setup.debug
6 drwx 512 Dec 29 2014 17:06:08 +08:00 c2960x-universalk9-mz.150-2.EX5
645 drwx 512 Dec 29 2014 17:06:09 +08:00 dc_profile_dir
648 -rwx 3763 May 25 2024 15:57:54 +08:00 private-config.text
651 -rwx 7976 May 25 2024 15:57:54 +08:00 config.text
652 -rwx 6168 May 25 2024 15:57:54 +08:00 multiple-fs