Cisco catalyst 3650 memory high

anhtuan.tran@gtc-jsc.vn · ‎08-25-2019

Help me!

Log Buffer (4096 bytes):

Interface EOBC18/1, changed state to up

*Aug 23 04:55:41.364: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down

*Aug 23 04:55:41.364: %LINK-3-UPDOWN: Interface LIIN18/2, changed state to up

*Aug 23 04:55:41.500: WCM-PKI-SHIM: buffer allocation failed for SUDI support check

*Aug 23 04:55:41.500: PKI/SSL unable to send Sudi support to WCM

*Aug 23 04:55:41.863: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!

*Aug 23 04:55:31.724: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is nocable

*Aug 23 04:55:31.724: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is down

*Aug 23 04:55:31.724: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is nocable

*Aug 23 04:55:31.724: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.

*Aug 23 04:55:31.724: %STACKMGR-6-ACTIVE_ELECTED: Switch 1 R0/0: stack_mgr: Switch 1 has been elected ACTIVE.

*Aug 23 04:55:41.879: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP

*Aug 23 04:55:42.392: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi18/3, changed state to up

*Aug 23 04:55:42.401: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC18/1, changed state to up

*Aug 23 04:55:42.401: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down

*Aug 23 04:55:42.401: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN18/2, changed state to up

*Aug 23 04:55:44.669: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

*Aug 23 04:55:48.277: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-vrf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001

*Aug 23 04:55:49.295: %SYS-5-CONFIG_I: Configured from memory by console

*Aug 23 04:55:50.003: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan44, changed state to down

*Aug 23 04:55:50.356: %SYS-5-RESTART: System restarted --

Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.5, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Mon 10-Dec-18 11:34 by mcpre

*Aug 23 04:55:50.499: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Aug 23 04:55:50.888: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down

*Aug 23 04:55:51.182: %SNMP-5-COLDSTART: SNMP agent on host ToR-S108 is undergoing a cold start

*Aug 23 04:55:52.861: %SYS-6-BOOTTIME: Time taken to reboot after reload = 380 seconds

*Aug 23 04:55:56.061: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)

*Aug 23 04:55:58.711: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP

*Aug 23 04:56:10.996: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: EDAC DEVICE1: CE: octeon_l2c_err instance: l2c0 block: l2c1 'L2D Single bit error (corrected):type:2, syn:0x212, way:127943'

*Aug 23 04:56:12.002: %IOSXE-4-PLATFORM: Switch 1 R0/0: kernel: EDAC DEVICE1: CE: octeon_l2c_err instance: l2c0 block: l2c1 'L2D Single bit error (corrected):type:2, syn:0x212, way:127943'

*Aug 23 04:56:22.197: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Gi1/1/1

*Aug 23 04:56:22.203: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Gi1/1/2

*Aug 23 04:56:24.656: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up

*Aug 23 04:56:25.656: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up

*Aug 23 04:56:27.004: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan44, changed state to up

ToR-S108#

Version

Switch Ports Model SW Version SW Image Mode

------ ----- ----- ---------- ---------- ----

* 1 28 WS-C3650-24TD 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE

Leo Laohoo · ‎08-26-2019

What is the question?

Giuseppe Larosa · ‎08-26-2019

Hello Anhtuan,

you have 99% of switch memory in use !!!!

The switch is likey affected by a SW bug causing memory leakage.

In the mean while schedule a reboot and ask for a maintenance window as soon as possible.

The memory can become so little free that the system will not be able to create a VTY session.

The image2.png attachment contains the show proc memory sorted.

*Init* is the process using more memory.

Pool Manager has a lot of activity.

As explained in this bug description it is the 5th column Holding that signal a process not releasing memory.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp57820

Hope to help

Giuseppe

giovannibardetta · ‎05-11-2021

Hi Have the same problem on 3850 with XE Everest 16.6.2

2021-05-05 00:01:13 Local3.Error 10.137.245.6 6701: *May 5 00:11:52.229 Italy: %PLATFORM-3-ELEMENT_CRITICAL:Switch 1 R0/0: smand: 1/RP/0: Used Memory value 98% exceeds critical level 95%

Leo Laohoo · ‎05-11-2021

@giovannibardetta wrote:

2021-05-05 00:01:13 Local3.Error 10.137.245.6 6701: *May 5 00:11:52.229 Italy: %PLATFORM-3-ELEMENT_CRITICAL:Switch 1 R0/0: smand: 1/RP/0: Used Memory value 98% exceeds critical level 95%

We have already responded to your thread.