Weird Issue with port channel on cisco 3560 switch connected to ESXi servers

[piyushewkani]

Hi,

I am facing a weird issue in setting up port channels between cisco 3560 switch and ESXi server. I have attached a file to refer the topology.

I have created this topology in my lab, where i have a Desktop port connectivity available from my university;s network. I wanted access to this lab via VPN/wifi hence i connected my switch to the Dports in my lab. I have been a two vlans (125 and 1226) which are enabled on two dports respectively. I connected one interface of my switch to the Dport giving access to vlan 125 and another interface to the Dport giving access to vlan 1226. I have configured both these interfaces as Trunk ports with native vlan set as vlan 125 and vlan 1226 respectively. (I know access --> trunk configuration doesn't works but i have been using this set up since long time and it fulfills my purpose as this is just a lab environment.) 

I connected two ESXi 6.5 servers having 5 nics each to the switch as shown in the topology. vmnic 0 and 1 of both servers connect to access port vlan 1226 and vmnic 2 and 3 connect to access port vlan 125. 

I have configured port channels on the interfaces which connect to ESXi servers and configured nic binding on ESXi vswitches (vswitch0 for vlan 1226 network and vswitch1 for vlan 125 network). 

I first made the port group 1 up and verified the connectivity to vms running on ESXi1 which is working perfectly fine (getting ip in vlan 1226 range). I then, brought po2 up, connecting to ESX1 vmnic 2 and 3. Verified the conectivity, which works completely fine. (VMs getting ip range in vlan 125 network, accessible from my university's wifi/vpn).

Then I brought po3 up connecting to ESXi2 vminc 0 and 1, which worked as it should (VMs running on ESXi2 getting ip range in vlan 1226 network range). The moment i bring po4 (connecting to ESXi2 vmnic 2 and 3 in vlan 125) up, i see this weird alternate flapping of my trunk ports (fa0/1 connecting to Dport vlan 1226 and Fa0/33 connecting to Dport vlan 125).

The uplink trunk port fa0/33 first goes down, then after a while it comes back up and the very next moment Fa0/1 trunk port goes down and this keeps repeating. When i shut the po4, the flapping stops.

Can you please suggest me how to approach in troubleshooting this issue? 

Here the is configuration of trunk, access and port groups

interface FastEthernet0/1
 description TRUNK-TO-VLAN-1226-DPORT
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1226
 switchport trunk allowed vlan 1226
 switchport mode trunk
 spanning-tree bpdufilter enable
end
!
interface FastEthernet0/33
 description TRUNK-TO-VLAN-125-DPORT
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 125
 switchport trunk allowed vlan 125
 switchport mode trunk
 spanning-tree bpdufilter enable
end
!
!
interface Port-channel1
 description ESXi1-PORT-GROUP-VSWTCH--FA0/2-FA0/17
 switchport access vlan 1226
 switchport mode access
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
interface FastEthernet0/2
 switchport access vlan 1226
 switchport mode access
 channel-group 1 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
!
interface FastEthernet0/17
 switchport access vlan 1226
 switchport mode access
 channel-group 1 mode on
 spanning-tree portfast
 spanning-tree bpduguard enable
end
!
interface Port-channel2
 description ESXi1-PORT-GROUP-VMGMT-FA0/18-&-FA0/19
 switchport access vlan 125
 switchport mode access
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
interface FastEthernet0/18
 switchport access vlan 125
 switchport mode access
 channel-group 2 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
!
interface FastEthernet0/19
 switchport access vlan 125
 switchport mode access
 channel-group 2 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
!
interface Port-channel3
 description ESXi2-PORT-GROUP-VSWTCH0-FA0/5-&-FA0/9
 switchport access vlan 1226
 switchport mode access
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
interface FastEthernet0/5
 switchport access vlan 1226
 switchport mode access
 channel-group 3 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end

DCND-MGMT#sh run int fa0/9
Building configuration...

Current configuration : 168 bytes
!
interface FastEthernet0/9
 switchport access vlan 1226
 switchport mode access
 channel-group 3 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
interface Port-channel4
 description ESXi2-PORT-GROUP-VMGMT-FA0/20-&-FA0/21
 switchport access vlan 125
 switchport mode access
 spanning-tree portfast
 spanning-tree bpdufilter enable
end

interface FastEthernet0/20
 switchport access vlan 125
 switchport mode access
 channel-group 4 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end
!
interface FastEthernet0/21
 switchport access vlan 125
 switchport mode access
 channel-group 4 mode on
 spanning-tree portfast
 spanning-tree bpdufilter enable
end




DCND-MGMT#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 8
Number of aggregators:           8

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)          -        Fa0/2(P)    Fa0/17(P)
2      Po2(SU)          -        Fa0/18(P)   Fa0/19(P)
3      Po3(SU)          -        Fa0/5(P)    Fa0/9(P)
4      Po4(SD)          -        Fa0/20(D)   Fa0/21(D)

[piyushewkani]

Missed the attachment. Please find it here

[chrihussey · Accepted solution]

Suggest disabling all the other port channels and then bring up just Po4.

     - If the problem occurs then it is isolated to this connection. For starters, Try bringing up only one link of the port channel, then take it from there.

     - If there are no issues then bring up Po3. If it starts happening then you have isolated it to something with ESXi2. Possibly something in the ESXi configuration is causing an L2 loop. (That appears to be what is happening anyway).

     - If Po3 & 4 are up and no problems, then bring up Po1. If it starts happening, shut Po3. If it is still happening then it has something to do with VLAN 125.

     - If everything works until the fourth port channel (Po2) is brought up it will require a deeper dive.

Lastly, just and observation, no exactly sure why the connections to the Dports in the lab are configured as trunks. With allowing one VLAN and making that VLAN native you are essentially accomplishing the same thing as if it were an access port.

Regards

[piyushewkani]

Hi Chris,

First of all, Thank you very much for looking into this issue and taking some time out to provide your expertise. Really appreciate it!

I followed your suggestion of bringing the port channels up one by one in order to isolate which port channel/interface is causing the problem. After disabling all the port channels and just bringing the po4 up, the trunk interfaces started flapping.  As you mentioned, this meant that the issue is specifically with po4, so I focused on configurations at both ends (switch and ESXi vswitch nic teaming config). 

I tried bringing only one link in po4, that caused the flapping so I felt maybe it's not the switch configuration which is causing a problem. 

I verified the vswitch1's configuration for nic teaming and it was similar to ESXi1's settings. I read few blogs on vswitch troubleshooting and someone suggested deleting and re-creating the vswitch. In order to that, I had to first power off all the VMs running on ESXi2 connected to vswitch2(Vlan 125). So I went ahead to powered them off. Before re-creating the vswitch I just thought of bringing the po4 back up again and see if this changes anything and voila! The flapping did not happen again! 

This came to me as a surprise, That this issue was definitely because of one of the (or many) VMs running on vswitch1. Just to give you a background, there were few OpenStack controller and compude nodes running on ESXi2 before this issue occurred. The students in the lab were trying to implement OpenStack and they were asked to configure static IPs. I can't think of what can cause this strange flapping issue of the trunk ports? Any ideas?

I am gonna try to bring each VM up one by one and try and see if this issue occurs again. If it does, it will lead me to the root cause. 

Regarding your query about the trunk interface configuration, I know the trunk configuration looks unconventional but this is what worked for us in the lab. Since the Dports are normal desktop LAN ports, they are configured as access ports. firstly, I tried simply connecting the interface to dport without any configuration (just a switch port), it didn't work (It should've, I am wondering why it didn't?). The VMs which eventually connected couldn't get the IPs (DHCP timeout was observed in captures). I knew access mode will not work, so trunking was the only option. Setting native VLAN to 125/1226 worked. Allow VLAN command is not needed, it works even without that. I just haven't removed it as it doesn't matter. 

Once again, Thank you very much! Really appreciate your inputs, knowledge and the support you provide to the community! 

Regards,

Piyush

[chrihussey]

Glad to hear things are working. Thanks for the update and votes.

I don't do much on the VM side of things so really can't speak to why one would cause this issue. However, it does sound like whatever it is was causing a layer 2 loop or possibly some form of broadcast storm.

As far as the trunk interface configuration is concerned just two things:

1- You said you tried to connect the dport to the switch port without any configuration. I may have misunderstood, but was it configured as an access port in the proper VLAN?

!

interface FastEthernet0/1

switchport mode access

switchport access vlan 1226

!

2- If it only works with the trunk configuration suggest keeping the allowed VLAN statement. Without it all broadcasts and miscellaneous LAN chatter from other VLANs will propagate to the dport. They'll probably just get discarded, but it is noise you just don't need.

Regards

[Rick1776]

It's also suggested to not have any interface configurations under the physical port that is part of the Port-Channel. I've seen many issues with this in IOS and NX-OS type switches. All you would need is the no shut and port-channel XXX commands and configure all the options under the Port-Channel.