Certain Devices drop off network without sticky mac enabled 2960-X

[Thomas Johnston]

Hello,

I have a bizarre issue with 2960-X models (WS-C2960X-48FPS-L). After upgrading to any release after 152-4.E8.bin certain devices drop off the network periodically. This can range from every 10 minutes to around 4 times a day. The devices most affected by this are Panasonic security cameras. The only fix I can find is to enable sticky mac address on the ports the cameras are connected to. Once this is in place, the issue goes away and the camera stays connected. This also affect certain things like security door nodes and other "dumb" devices. 

Current version of IOS:152-7.E4.bin
Interface configuration:

 switchport access vlan xxx
 switchport mode access
 switchport voice vlan xxx
 switchport port-security maximum 10
 switchport port-security aging time 5
 switchport port-security
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 snmp trap mac-notification change added
 snmp trap mac-notification change removed
 mls qos trust device cisco-phone
 mls qos trust cos
 auto qos voip cisco-phone
 spanning-tree portfast edge
 service-policy input AUTOQOS-SRND4-CISCOPHONE-POLICY
 ip dhcp snooping limit rate 100

 Below where the arrow is I applied sticky mac to the interface. As you see the broken gray line before is where it was dropping.

Applying sticky mac to every interface isn't feasible in this environment as users are constantly moving around and I do not have time to be manually removing mac addresses from interfaces. I've also tried moving the aging time higher to something like 2 hours but that didn't help either.

-Devices do not lose link
-Devices do not lose PoE

They just simply stop responding and then magically come back.

Other than applying sticky mac the only fix so far is to stay on the older version of code and never upgrade which is bad for security.

Anyone have any ideas on what could be causing this?

[lagerplane]

This seems one of those situations where you want Wireshark to see what happens on the wire and see who is at fault.

When you said "They just simply stop responding and then magically come back" , when it is on the broken state, does the switchport stops seeing the MAC address of the end device? ,  and "stop responding" means you cannot ping the device right? (not even on the same subnet, say ping from the Default Gateway of the end device subnet).

[Thomas Johnston]

When I say not responding yes, no longer replies to ICMP or http requests. The MAC address is also gone from the table. When the device is reachable again, its back in the table. I'll need to break it again and see if I can capture it via wireshark. I think it also affected a few windows PCs but never was able to confirm that as I rolled the code back as it needed to be fixed. It was also so intermittent on the PCs that it was impossible to be around when it happened. Cameras need to be on 24/7 which trumps the security issues with the IOS code at this point. Seems like something with the mac aging out but can't get a handle on it. I do know the cameras do not record 24/7 and only on motion so that may have something to do with it. Though we do monitor them for uptime and they get pinged every 60 seconds. Though since this traffic originates from something other than the device itself, I think it can still age out.

[David Ruess]

Hello,

So with the aging command if there is not activity it will flush the MAC if its dynamically learned. With the sticky option you applied earlier it will keep it regardless. I would remove the aging and keep the sticky if you need constant connectivity.

Hope that helps

-David

[legacynetwork]

Strangely, I have more or less same issue. People in Wired/Wireless connection are dropping randomly, there is no pattern at all. I have C2960X STACK + ISE Authentication as well.