Core Stack Mistake == Looking for advice

[jdamon113]

Here is the overview. Two years ago went from 3com 5500 to cisco. Nice Move . 

environment is 50 - 60 servers ESXi running about 35 servers.  First wave was to replace the core stack, Which were 3 3com 5500G , This was okay until we starting adding more and more devices. This part was my bad, I took the advice of a Cisco sales rep and place Three 3850's with the 10G modules, (As my Core )Now all 3com's are gone and I have put in 2960xs for our phone and access switches. These are running great. I have around 23 of these in two location via 10g fiber from MDF to IDF> NICE )

All servers, p-2-p devices, FW, asa and several network aware appliances and a Full 3 host esxi envirment are connected to the core stack of the 3 3850's 

Each esxi host has 10 1G connections to the core pluse all other devices. I have almost no room left on the core maybe a total of 10-15 ports left between the three units.  My poor Core is always around 70-90% cpu utilization. But so far no outages, just slow points on some servers. I am unable to upgrade as the system is still very new . I have several 2960xr with the 10g modules, I was thinking about putting in two just for the esxi servers , but If I do that, I am still passing that traffic to the 3850's , Not sure of a better way to look at this. Please any thoughts ?  

[dbellaze]

What are the process's consuming 70-90% cpu? From a pure L2/L3 perspective I wouldn't expect such high cpu unless there is a lot of interrupt traffic. Is it possible you have some traffic looping?

[jdamon113]

Thanks Dbellaze 

I have check for such things and just in case I did not want to be bold and think I know everything, I have a cisco tac look, they noted all spanning tree is fine, no loops and that the 3850 switch could handle this load. IF your able to place in some prospective please feel free to do so .

For the record, I do not believe the cisco tac on this one, 70-90 %. Its too much. The stack is about 2 years old now. At this rate, it will only hold out for another two.  

Just as I was getting this file , I see this today. which is new Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi1/0/22 and port Gi3/0/12
Sep  5 17:33:53.763: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi1/0/22 and port Gi3/0/12 (CORESTACK-2)
Sep  5 18:43:53.426: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi3/0/3 and port Gi1/0/22
Sep  5 18:43:53.521: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi3/0/3 and port Gi1/0/22 (CORESTACK-2)
Sep  6 02:38:54.084: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi1/0/22 and port Gi3/0/12
Sep  6 02:38:54.210: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi1/0/22 and port Gi3/0/12 (CORESTACK-2)
Sep  6 03:48:24.693: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi1/0/22 and port Gi3/0/12
Sep  6 03:48:24.813: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi1/0/22 and port Gi3/0/12 (CORESTACK-2)
Sep  6 10:31:21.920: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi3/0/12 and port Gi1/0/22
Sep  6 10:31:22.059: %SW_MATM-4-MACFLAP_NOTIF: Host 90e2.ba03.00d1 in vlan 4 is flapping between port Gi3/0/12 and port Gi1/0/22 (CORESTACK-2)

Thanks for your input. 

[dbellaze · Accepted solution]

This article might aid in trying to dig into the cpu usage a bit more.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

From your log it does look like either there is a loop of some sort or that could be a duplicate MAC being used in the network. I would trace to the source/s and work your way outward looking at any redundant paths physical or logical.

Do you have a diagram of the physical topology? Also is STP running and is it stable or showing signs of topology changes? 

[jdamon113]

Thanks

I will trace that down, As for maps or diagrams, have not yet made them. Been a great year for projects.

Thank You much

[jdamon113]

I found it. Thanks for the help.

I have a bad multi port Ethernet handing out the same mac into the core.

[dbellaze]

Interesting so the cpu utilization is not high anymore? If you don't mind sharing more details that would be great!

[jdamon113]

Its down to around 40 - 50 during peak hours. Ill keep checking it though