Multiple MAC Adresses on the first Port (C1000 Stack) with only one device connected to it

[advuni-af1]

Hello Cisco community,

we have a strange problem with our new Catalyst C1000 infrastructure. We implemented 4 x Nexus 3524X for core, 9800CL and 9115 APs for WLAN, ISE for security and C1000 for wired client access. So far so good.

The infrastructure replaced a HP/ProCurve network and now we have problems which didn't exist with the old network.
These are for example:
- VoIP calls which terminate unexpectly or the one person can't hear the other but pour the other can.
- Clients disconnects from the network unexpectly and don't reconnect.
- Autonegotioation works unreliable

All these problems are in context to the Catalyst C1000. We have implemented round about 28 switches of the C1000-48P-4X-L.
The most switches of them are in 2 switch stacks over the 10 Gbps ports (HStack setup). Then the uplink is a portchannel of 2 x 10 Gbps from stack member 1 and 2 to the Cisco Nexus 3524X which are in a vPC pair.
All C1000 have the same software version: IOS 15.2(7)E3k. We run into a BUG (CSCvv48770) after transition why we musted update to this version in January.
However there are always problems (see above) which we can't find and fix. Also logging and so on didn't deliver a solution for this.

Anyway, but what we can see, the C1000 have multiple MAC Adresses on the first port where only one device is connected. This is very dynamicly. At the moment you see 12 MACs for example, 5 seconds later this are 8, then again 12 and so on.
And this alo for VLANs which don't have any port connected on these switch. We think, it could be the HStack Setup? We also breaks the stack ring. This didn't change anything. All of the ports are configured as "normal" access ports on the switch.

I attached a screenshot of the behavior. Also I will post the config of the ports.

Does anyone have similar problems? Or can check it on his infrastructure?

Best regards

Alex...

[advuni-af1]

Here are two examples of the behavior and the config of this port.

This is definitely a problem of the switches. Here is a CLI output of two of the switch stacks.

Example A:

b1-og3-01#show mac address-table interface gigabitEthernet 2/0/1
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
120 3007.4df7.3303 DYNAMIC Gi2/0/1
120 f8ac.65bf.3961 DYNAMIC Gi2/0/1
160 7478.2759.7700 DYNAMIC Gi2/0/1
160 9ceb.e8c4.f4af DYNAMIC Gi2/0/1
Total Mac Addresses for this criterion: 4

Example B:

b2-ug-01#show mac address-table interface gigabitEthernet 2/0/1
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
8 001a.e87b.a581 DYNAMIC Gi2/0/1
8 001a.e87b.dd80 DYNAMIC Gi2/0/1
104 5488.ded9.f1bc DYNAMIC Gi2/0/1
128 bc3d.85c4.9fcf DYNAMIC Gi2/0/1
161 001a.e87b.dd80 DYNAMIC Gi2/0/1
Total Mac Addresses for this criterion: 5

We checked the circumstances several times. Only one device is connected to the port and the port is only in two VLANs (Client/VoIP).

b1-og3-01#show derived-config interface gi 2/0/1 Building configuration...

Derived configuration : 348 bytes
!
interface GigabitEthernet2/0/1
description vlan_160_client_b1
network-policy 1
switchport access vlan 160
switchport mode access
switchport block unicast
load-interval 30
storm-control broadcast level pps 1k
storm-control multicast level pps 2k
storm-control action trap
spanning-tree portfast edge
ip dhcp snooping limit rate 15
end

b2-ug-01#show derived-config interface gi 2/0/1 Building configuration...

Derived configuration : 348 bytes
!
interface GigabitEthernet2/0/1
description vlan_161_client_b2
network-policy 1
switchport access vlan 161
switchport mode access
switchport block unicast
load-interval 30
storm-control broadcast level pps 1k
storm-control multicast level pps 2k
storm-control action trap
spanning-tree portfast edge
ip dhcp snooping limit rate 15
end

The MAC Addresses on this ports changes dynamicly. Some times you see 3 or 4, othertime you see 10 or so...

[Richard Burts]

What device is connected to these ports? If it were some VM, or some switch, or some access point then multiple mac might be expected.

[advuni-af1]

Hello Rick, that's the point. Only one device (PC/Laptop) is connected to these ports. I checked it several times on several switch stacks. When there are APs with FlexConnect or another switch then it's normal to have multiple MACs on these ports. I think, it's a bug in the IOS for this switches. But Cisco didn't help me because the devices aren't covered with SNTC. And then TAC says: Not our problem. But the problem is definitly on the switches/software.

[Richard Burts]

Alex

Sorry that TAC was not more helpful. I am not optimistic that we will be able to figure out much, but I do have several observations and questions:

- in the original post the example you provided was 1/0/1, and in a following post you gave 2 examples both of which were 2/0/1. Does this always happen on x/0/1 interfaces? Or does sometimes it happen on interfaces beyond /0/1?

- In each of the examples you provide the mac addresses shown are associated with multiple vlans. In the second set of examples it was 2 vlans and the first example was MANY vlans. But it looks like the interfaces involved are configured as access ports. It seems to me that this does point at it being an issue in the C1000 code.

- If does look like a bug, perhaps a different version of code might improve things?

[advuni-af1]

Hi Rick,

thanks for your answer.

 - On all switch stacks these are always the first ports. How you wrote gi X/0/1. It doesn't matter if this is the stack master or standby.

 - This is one of the big problems. The ports are only in the client vlan and with a network policy in the voice vlan. From config perspective this are normal access ports. I don't understand why the mac address table shows macs from vlans on these ports which are not assigned to this ports. Really strange.

 - We have IOS 15.2(7)E3k running. I could do an update to IOS 15.2(7)E4. But the release notes didn't say anything to a problem that even comes close to this.

 - We will buy SNTC and reopen the case. When I have a solution I will post it here.

BR Alex...

[Richard Burts]

Alex

Thanks for the additional information. If it always happens on the first port of the switch (and not ever on higher numbered ports) then I would regard that as a strong indicator that this is a software bug.

If we believe that it is a software bug in that version of code then a different version "might" solve the problem. Certainly no guarantees. If release notes do indicate that some particular bug if fixed in version x then this would be a good reason to upgrade to that version. If the release notes do not mention a fix for that bug it does not necessarily mean that the release would not affect it - especially in the case of something where you can not find an identified bug that creates the particular symptoms you are experiencing.

I hope that you are able to reopen the case and that TAC does find something. Please keep us updated. 

[Daviid Kumer]

Hello everyone,

I have the same problem as described, a C1000 aggregation switch is constantly reporting MAC flapping of a client between two access switches connected to the aggregation switch on separate copper/optical links, bundled in etherchannel. On the access port where the MAC address flapping is originating i have a Cisco IP phone and a PC, and the aggregation switch is reporting MAC flapping between two switches that are installed in physically different buildings (so no logical way i would have a loop). In my case, a shutdown of an interface in the port channel (two optical links) leading to the switch in the remote building has completely eliminated MAC flapping, but i have to shutdown a specific interface to achieve this (shutting down the second interface in the same port channel did not resolve the port-flapping). If anybody has found a solution for this, i would kindly ask you to share the resolution.

Thanks

Br

David

[Georg Pauwen]

Hello,

when the flapping occurs, does the MAC address in question show up on the port channel (show  mac address-table) or on the physical interface(s) ?

[Daviid Kumer]

From the perspective of the aggregation switch, the MAC flapping occurs between the two port-channels connecting to access switches in different buildings. The only difference is one is connected via copper links, the other via optical links.

[Daviid Kumer]

From the perspective of the aggregation switch, the MAC flapping occurs between the two port-channels connecting to access switches in different buildings. The only difference is one is connected via copper links, the other via optical links.