%% Low on memory; try again later

[star1]

Hello guys,

Recently i have lost remote connections to one of my access 9300 switch, i cannot SSH it, when I consoled to the switch it didn't allow or prompt me to put the user name and password, it showed lots of error regarding the memory low.

I am thinking to reboot the device which it might sort the problem but before I do that, i was hoping to get more info before, I am worried that the switch might not come back up after the reboot.

any suggestion I appreciated.

The switch model and error are below.

Cisco Catalyst 9300 Series Switches
Software Image: 16.9
from the DNA I can see this error, Partial Collection Failure (Connection Error)

Note the switch manged through DNA.

% Low on memory; try again later

%% Low on memory; try again later

Feb 13 2020 10:59:26.089 UTC: %SISF-3-INTERNAL: Internal error, Cannot chunk malloc mac_entry -Process= "SISF Main Thread", ipl= 0, pid= 351
-Traceback= 1#d6191b37f28bb99e33629b4a892bf690 iosd_shr_m_uk9_IPV6_Core_crb:7F673D2E9000+3B2F5E iosd_shr_m_uk9_IPV6_Core_crb:7F673D2E9000+34237D iosd_shr_m_uk9_IPV6_Core_crb:7F673D2E9000+3353E8 iosd_shr_m_uk9_IPV6_Core_crb:7F673D2E9000+3782AB iosd_shr_m_uk9_IPV6_Core_crb:7F673D2E9000+3B7435 :560D97302000+68F4D6D iosd_shr_m_uk9_IPV6_Core_crb:7F673D2E9000+3B5F82 :560D97302000+68CA8A6
Feb 13 2020 10:59:44.469 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 32772 bytes failed from 0x8000560D9DA65D48, alignment 8
Pool: Processor Free: 1942624 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

Thank you in advance

[Georg Pauwen · Accepted solution]

Hello,

a reboot will probably help, what is the uptime of the switch ?

Also, do you have device tracking enabled on one or more interfaces ? I found the bug below for version 16.9., it doesn't really look related though...

C9300 on 16.9.3 may log the error: %SISF-3-INTERNAL: Internal error, Unknown af 0x7F7D3B7F3A14

CSCvr89684

Description

Symptom:
The following error will be seen in logs: %SISF-3-INTERNAL: Internal error, Unknown af 0x7F7D3B7F3A14, 0 -Process= "SISF Main Thread", ipl= 0, pid= 349

Conditions:
C9300 running 16.9.3 with SISF-based device tracking configured

Workaround:
N/A

Further Problem Description:

Customer Visible

Notifications

Save Bug

Open Support Case

Was the description about this Bug Helpful?

(0)

Details

Last Modified:

Dec 6,2019

Status:

Open

Severity:

4 Minor

Product:

(1)

Cisco Catalyst 9300 Series Switches

Support Cases:

0

Known Affected Releases:

(1)

16.9.3

[star1]

Hello Georg,

Thank you for the quick reply,

Answer your questions.

Uptime: 15 days 9 hours 23 minutes.

Not sure if there will be  any of the interfaces been enabled as tracking interface, I only have access to the switch through the DNA, which is not very helpful as I am CLI person. :-(

Is there any way I can check it through the DNA please?

Regarding the bug, do you think if it was the case, it would have the same impact to all other switches?

Thank you in advance

[star1]

Hello,

Reboot sorted the problem. thank you. 

[Joseph W. Doherty]

"Cause: Memory fragmentation"

To my knowledge, Cisco network devices don't have a garbage collector. I.e. they can "run of of memory", when actually there's much available, but also much fragmented. Reloads are often the only cure.

To preclude this happening, you can insure there always a large amount of available memory. If something is consuming memory, try to discover what it is, and see if a configuration change might reduce memory consumption. (If also might be due to a memory leak, then the preclusion often entails a new IOS with the memory leak bug fixed.)