Fixes for weird dual homed vPC FEX - Issues where it will not take configuration:
We had issues with a Dual Homed vPC Multi FEX install head-ended by two N93180YC-FX3 in a vPC.
After having a failed N93180-FX3, we had to replace it and reassociate it to the 5 other FEXs and the N93180-FX3 PEER.
We upgrade firmware/epld on the replacement N93180-FX3 and uploaded the backup config to switch and were able to get
all but the FEXs configured. We attached a temporary FEX to the N93180-FX3 to allow us to provision the FEX interface
configurations, but this caused issues when we moved the temporary FEX to another association (102,104,106,108,110 etc).
We gave up with the lab testing/build and proceeded to move this into production.
We then moved the N93180-FX3 into production and connected all interfaces except the FEXs (keepalive, peerlinks, trunks to core, etc).
All interfaces and portchannels came online, the vPC formed, and all looked good. We then moved one FEX interface and brought it online.
Interface came up, fex associated, and all existing connections to our other N93180-FX3 continued to function. We then programmed
the interfaces for this FEX (Ethernet106/1/1-48) using backup configurations.
All went well except when we moved 102 (temporary FEX we configured before putting into production). No matter what we tried this FEX
would not take configurations (it would accept in the config term mode with no errors, but when issued a `sh run int e102/1/1` the configuration
would not be consistent - Mismatched options shutdown, no desc, etc)
We fought with this for a while and tried defaulting the interfaces associated with that FEX with no effect.
As a last ditch effort removed the FEX and its associations with the FEX-fabric. This allowed us to finally get the FEX to accept
the configurations.
**Note that during this change we still had connectivity from the FEX in question to the other vPC N93180-FX3 and did not loose link or flows from
this FEX.**
Hopefully this helps someone in a bind!
Below is a config snippet of the changes applied to the secondary vPC member to get the FEX 102 configured:
DCDIST-N9K-2# conf t
Enter configuration commands, one per line. End with CNTL/Z.
DCDIST-N9K-2(config)# no fex 102
ERROR: Error: Fabric ports exist for this FEX
!!
!!Cant remove the fex as it is associated with a fabric interface
!!
DCDIST-N9K-2(config)#
DCDIST-N9K-2(config)# sh run int po102
!Command: show running-config interface port-channel102
!Running configuration last done at: Mon Dec 18 10:01:51 2023
!Time: Mon Dec 18 10:14:32 2023
version 10.2(5) Bios:version 01.08
interface port-channel102
description FEX-102
switchport
switchport mode fex-fabric
fex associate 102
vpc 102
DCDIST-N9K-2(config)# conf t
!!
!! Go to the portchannel associated with the FEX you want to remove
!!
DCDIST-N9K-2(config)# int po102
!!
!! Remove the FEX association on the port-channel
!!
DCDIST-N9K-2(config-if)# no fex associate 102
DCDIST-N9K-2(config-if)# exit
!!
!! Now remove the FEX config
!!
DCDIST-N9K-2(config)# no fex 102
!!
!! Now recreate the FEX config
!!
DCDIST-N9K-2(config)# fex 102
DCDIST-N9K-2(config-fex)# pinning max-links 1
Change in Max-links will cause traffic disruption.
DCDIST-N9K-2(config-fex)# description "FEX0102"
!!
!! Go to the portchannel associated with the FEX you want to add
!!
DCDIST-N9K-2(config-fex)# int po102
DCDIST-N9K-2(config-if)# fex associate 102
DCDIST-N9K-2(config-if)# exit
!!
!! Wait for FEX to come online and then program the ports
!!
DCDIST-N9K-2(config-if)# interface Ethernet102/1/1
DCDIST-N9K-2(config-if)# description <Description here>
DCDIST-N9K-2(config-if)# switchport
DCDIST-N9K-2(config-if)# switchport access vlan <vlan here>
DCDIST-N9K-2(config-if)# no shutdown
DCDIST-N9K-2(config-if)#
DCDIST-N9K-2(config-if)# interface Ethernet102/1/2
DCDIST-N9K-2(config-if)# description <Description here>
DCDIST-N9K-2(config-if)# switchport
DCDIST-N9K-2(config-if)# switchport access vlan <vlan here>
DCDIST-N9K-2(config-if)# no shutdown
DCDIST-N9K-2(config-if)#
!! ... ETC ... ETC
