cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3164
Views
0
Helpful
5
Replies

Catalyst 3750 low on memory

jackson.ku
Level 3
Level 3

Hi,

I have 3 Catalyst 3750 in same stack, the IOS version is 12.2(53)SE2. Today we can not telnet/ssh to this switch, but ping is ok, and switch function is ok. I try to access the console port, it show "low on memory, try again later". After I reboot the master switch in the stack, the master switch change to another switch, then I can telnet/ssh to this switch.

I check the Ciscoworks server syslog report, there are many MAC address flapping error message, and I beleive the MAC address flapping occured before several weeks. ( the G2/0/15 & G3/0/15 is connect to a VMware ESX server and the EtherChannel config mismatch with this Catalyst 3750 switch ) Please help to mention me the root cause of the "low on memory" problem and what is the abnormal memory usage ( free memory percentage below ?% )?

Best Regards,

some of MAC address flapping message :

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:47,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:47,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:47,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0006 in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:05,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:11:51,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:11:51,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0062 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:58,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:58,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:58,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:38,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:28,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:28,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi3/0/15 and port Gi2/0/15,*

172.16.51.97,172.16.51.97,Jul 26 2012 09:10:17,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*                 

172.16.51.97,172.16.51.97,Jul 26 2012 09:12:47,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:47,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:47,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:18,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0006 in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:12:05,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:11:51,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:11:51,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0062 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:11:40,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:58,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:58,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:58,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:38,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:28,SW_MATM,4,MACFLAP_NOTIF,Host 0050.569e.0050 in vlan 38 is flapping between port Gi2/0/15 and port Gi3/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:28,SW_MATM,4,MACFLAP_NOTIF,Host f4ce.4685.13d4 in vlan 20 is flapping between port Gi3/0/15 and port Gi2/0/15,*
172.16.51.97,172.16.51.97,Jul 26 2012 09:10:17,SW_MATM,4,MACFLAP_NOTIF,Host 000c.2921.53fe in vlan 38 is flapping between port Gi3/0/15 and port Gi2/0/15,*

5 Replies 5

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello Jackson,

the two issues are likely unrelated.

MAC address flapping may be a sign of a bridging loop occurring somewhere in your network. This can lead, for experience, to high cpu usage not to high memory usage.

Investigate on what is connected to the involved ports Gi2/0/15 and Gi3/0/15. it may be wise to disable one of the two ports to see if all this MAC flapping stops.

Configure correctly the etherchannel if it is configured on ESX side you need to configure also on switch side.

Later you can investigate the low memory issue that might be caused by a SW bug (a memory leak is the term used for this kind of scenario, if this is the case the low memory appears after N hours of uptime).

Hope to help

Giuseppe

 have 3 Catalyst 3750 in same stack, the IOS version is 12.2(53)SE2. Today we can not telnet/ssh to this switch, but ping is ok, and switch function is ok.

Sorry Giuseppe, I'll have to disagree with you.

I've seen this error message of "low memory" before.  And I've seen it on THIS IOS version.

Upgrade your IOS to, say, 12.2(55)SE5.

Because you can't "get in" to your switch stack, you may need to reboot the entire stack. 

Hello Leo,

my guess was the same a SW bug causing ,memory leak on the stack

. I had this issue on 12.2(25)SEE some years ago.

What I tried to say is that likely the "low memory" and MAC address flapping are unrelated. It was MAC address flapping the trigger of memory leak in your case?

In any case the good news is that you are suggesting a candidate release for IOS upgrade of the stack

Hope to help

Giuseppe

What I tried to say is that likely the "low memory" and MAC address flapping are unrelated. It was MAC address flapping the trigger of memory leak in your case?

Hi Giuseppe,

No MAC address flapping.  Just the darn code. 

skarthic
Cisco Employee
Cisco Employee

Jackson,

We will have to understand how the router got into low memory -

i) whether it was an instantaenous depletion  --> network events

ii) did this occur over a period of weeks/months --> normally due to bug

iii) or there could be a genuine usage of memory  --> check the config

For cases ii) and iii) checking the 'show mem stat' might help. Can you let us know when did you see MAC flaps previously?  What else was going on during the time the switch went into low memory?

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: