catalyst 4500 host flapping

[whanson]

Curious problem. We have 3 UPS connected to a catalyst switch via an ethernet. It would appear that the UPS is reflecting packets back to the catalyst such that the catalyst thinks that devices have moved to the port where the UPS is. Has anyone seen this??

[Kevin Dorrell]

Are these UPS linked together in any way apart from through the switch? It sounds like they are repeating the broadcasts back into the switch, maybe on each other's ports. But if that were so, the Spanning Tree would take care of it. You are right: it is a curious problem.

Are you getting any excessive broadcast storms?

Kevin Dorrell

Luxembourg

[Prashanth Krishnappa]

If the switch learns same unicast MAC address on multiple ports, it would log a host flap message. This is expected behavior and STP would not protect against this. I'll let somebody who might have experience with UPS like you have reply why the UPS would do this

[Kevin Dorrell]

Maybe I explained that badly. I was speculating that maybe the UPS were stacked together in some way. Of course the stack should only be connected to the switch at one point. But if you connected all three to the switch, it would be like connecting three ports to a hub. You would get a terrible broadcast storm for a moment, but then that is the situation the STP would take care of.

Unusual symptoms suggest unusual hypotheses.

Kevin Dorrell

Luxembourg

[milan.kulik]

Interesting...

I've seen such a situation.

But the reason was a hub with an MDIX auto port looped into another port.

Is there any hub or cheap (non-STP) switch involved?

Have you tried to configure bpduguard on the ports connecting the UPSs?

Are all of the 3 UPSs behaving the same way?

Regards,

Milan

[Kevin Dorrell]

What is the manufacturer and model of these UPS?

[tim.giles]

Did anyone get to the bottom of this problem??

Interestingly, we had a similar issue yesterday with a 2 x 3850 switch stacks we installed for a new customer. We've rolled out approximately 20+ of these stacks already and these are the first we've had problems with. The 3850's have replaced some Nortel 5510 stacks that were already in situe.

We spent a while troubleshooting the fault but ended up raising a TAC case due to the severity of the problem.

After going through 2 TAC engineers, the 3rd quickly found a problem with the MAC learning on the access port connected to a UPS (APC smart UPS 3000). It appears the UPS had the same MAC address as the core device connected to the 3850, hence the problem! It appears that the port is learning various MAC addresses from the connected UPS...there were various errors in the log:

Feb 12 11:21:26.977: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address xxxx.xxxx.xxx on port x/x/x

On one stack the access port connected to the UPS was shut down, solving the problem, on the other stack the cable connected to the UPS was disconnected, again problem solved.

TAC are saying the issue lies with the UPS learning bogus MAC addresses!

We have asked the customer for logs and configuration from the UPS, I will let you know if there's is anything of interest in them.

Anyone have any experience of this??

[tim.giles]

Hi,

Just an update on this in case anyone comes across the same issue again..it may save you a lot of pain!

We managed to get to the bottom of the problem and it does appear the UPS is reflecting packets back to the network. Cisco said they have never seen this with UPS' before but have with other devices that have faulty NIC's.

We identified the switchport was learning multiple MAC addresses from the UPS, not strange in itself, however it was when we discovered it was learning the MAC address of the core device, from the UPS! This meant traffic destined for the core was being black holed.

Anyway, to cut a long story short, I applied MAC security to the switchport so it only allowed the MAC of the UPS, and restricted the port so it only learns 1 MAC address (remove the maximum limit if you have one):

switch(config-if)#switchport port-security

switch(config-if)#switchport port-security violation restrict

switch(config-if)#switchport port-security mac-address abcd.abcd.abcd

Applying broadcast/multicast control is another best practice that may help also, i.e:

switch(config-if)#storm-control action

switch(config-if)#storm-control broadcast level

switch(config-if)#storm-control multicast level

switch(config-if)#storm-control unicast level

The model of UPS was an APC3000 with an old firmware version (1.1.5), none of the other UPS' in our customers estate have exhibited the same behaviour so we suspect it's only relevant to these devices. According to the UPS vendor, this firmware is old and needs upgrading...plus the devices are EoL!

We are going to implement port security on the UPS ports across the estate now, and put them into their own VLAN to mitigate against further issues with these type of devices.

I hope this helps some other poor soul who comes across this fault..