CIMC no accessible, monitoring free memory - FN - 63943

[stuartbedale]

Hi all

According to

Field Notice: FN - 63943 - Memory Leak in UCS-C Cisco Integrated Management Controller (CIMC) Might Cause Memory Exhaustion for Systems with Extended Up-Times - Software Upgrade Required

the work-around is

Until the software is released that corrects this issue, Cisco recommends that users monitor the free memory of their CIMC and plan a reload if the memory nears exhaustion. In order to avoid this issue, Cisco recommends that users upgrade their systems as shown in the tables that follow.

How can you monitor this on the CIMC, is there a Cisco MIB somewhere?

Thanks

[Niko Nikas]

Hi Stuartbedale,

Unfortunately I don't have a C-series in my lab to verify it is the same as a B-series.

However, you can monitor free memory from the B-series CIMC (which should be fairly similar to a C-series) if it's integrated into UCSM by connecting to that CIMC and then running 'memory'

[ help ]# memory

Memory Infomation

MemTotal:       113696 kB

MemFree:         83440 kB

Buffers:             0 kB

Cached:          15760 kB

SwapCached:          0 kB

Active:          15444 kB

For a standalone C-series, you'll need to essentially run the same thing.

SSH to the host and drop into the debug-shell, from there run 'memory'

[ help ]# memory

Memory Infomation

MemTotal:         111948 kB

MemFree:           33480 kB

Buffers:            1756 kB

Cached:            35836 kB

SwapCached:            0 kB

Active:            35708 kB

Hope this helps.

[stuartbedale]

Hello Ninikas

Thanks..on one of my lab boxes, I get this

ucs-c210-m1# scope cimc
ucs-c210-m1 /cimc # connect debug-shell
CIMC Debug Firmware Utility Shell [ cli ]
[ help ]# memory
Memory Infomation
MemTotal:       112944 kB
MemFree:         27592 kB
Buffers:          8076 kB
Cached:          38548 kB
SwapCached:          0 kB
Active:          49616 kB
Inactive:        24080 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       112944 kB
LowFree:         27592 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:          37060 kB
Slab:             6404 kB
CommitLimit:     56472 kB
Committed_AS:   178856 kB
PageTables:       1088 kB
VmallocTotal:   139264 kB
VmallocUsed:      3004 kB
VmallocChunk:   135676 kB
Load Infomation
1.10 1.27 1.26 2/166 18326
Up Time
52364398.98 646380.40
[ memory ]#

So I can get some numbers, is there a way to monitor this other than direct CLI commands.

The FN goes on to says you can reload as this approaches exhaustion, do you have any idea what level that might be considered to be?

Thanks.

Stuart

[Kirk J]

Greetings.

Rack server mib info: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/mib/c-series/b_UCS_Standalone_C-Series_MIBRef/b_UCS_Standalone_C-Series_MIBRef_chapter_0100.html

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/mib/c-series/b_UCS_Standalone_C-Series_MIBRef/b_UCS_Standalone_C-Series_MIBRef_chapter_011.html

 Download MIBs at: ftp://ftp.cisco.com/pub/mibs/supportlists/ucs/ucs-C-supportlist.html

The CIMC's memory information does not appear to be exposed as a SNMP object.

I filed an enhancement request, CSCvc97770, to have have snmp object created, as I have seen your question from other customers from time to time.

The manual way to check on the CIMC's memory is:

SSH to CIMC IP:

#connect debug-shell

#memory

If you stay below 20MB free, I would consider doing a CIMC reboot. 

The resulting output should include a field 'MemFree'.

If you have a M2 server, updating your HUU level past 1.43p, will avoid memleak.

If you have a M3/M4 server, updating your HUU level past 2.03i will avoid the memleak

Thanks,

Kirk...

[SamUrai]

I couldn't find a link to this FN.  Can someone please share it?  Thanks

[Steven Tardy]

Found FN63943 on an internal page which points to CDETS: CSCun88303 CSCus63934

Fixes for these bugs were released in 2014. If you are still running impacted firmware. . . . Why?
Go grab the latest/starred version.

[jameswood32]

1. Check CIMC Connectivity:

Ensure the CIMC IP address is correctly configured and reachable from the network. You can use a ping test or access the CIMC through its web interface.

Verify that the CIMC firmware is up to date. You may need to update the firmware if there are known bugs or compatibility issues.

2. Reboot the CIMC:

Sometimes, CIMC can become unresponsive due to internal errors. You can try rebooting it via the physical server or using a command from the system console.

You can reset the CIMC through the system BIOS or using the UCS Manager if you have access to it.

3. Check for Memory/Hardware Issues:

If the problem relates to monitoring memory, inspect if there are any hardware failures (e.g., faulty memory modules, CPU issues) reported in the CIMC logs.

Run diagnostics via the UCS Manager or directly from the CIMC interface to check if there are any hardware alerts or failures related to memory.

4. Look at CIMC and System Logs:

Review logs on the CIMC interface and the system console to find specific error messages or patterns that might indicate a more specific root cause.

In some cases, CIMC logs may show communication issues or failed memory monitoring attempts.

5. Update Monitoring Tools or Scripts:

If you are using an external monitoring tool or script to monitor free memory, verify that the tool is compatible with your hardware and CIMC version. Ensure the monitoring settings or SNMP traps are correctly configured.

6. Contact Cisco Support:

If none of the above steps resolve the issue, contacting Cisco Technical Assistance Center (TAC) would be the next step. Provide them with detailed information such as hardware model, CIMC version, and any logs that may help diagnose the issue.