vpc not passing traffic when one nexus switch is down

[sanchezeldorado]

Hello. I'm preparing to do some changes in my live environment and VPCs are new to me. I have a Cisco CML lab and I'm trying to setup a simple VPC configuration with two nexus 9000 series switches. I have a catalyst switch switch connecting an etherchannel to the VPC on one side, and a catalyst switch connection an etherchannel to the VPC on the other side. Both Catalysts have an SVI configured on my test vlan 526, and vlan 526 is allowed across the trunk link over my VPC. Etherchannels show up, configuration is identical on both nexus switches, and the VPC shows everything is up and running. If I shut down nexus switch 1, the network continues functioning without a problem. If I shut down nexus switch 2, all vlan 526 traffic stops. It doesn't matter if I change the vpc priority to change the primary. I'm at a loss for what more I can do. While nexus switch 2 is down, spanning tree looks normal, and the etherchannels are up while one of the links is down. Here's the relevant config:

Catalyst 1

interface Vlan526
ip address 10.10.52.2 255.255.255.0
!
interface GigabitEthernet2/2
switchport trunk allowed vlan 526,700
switchport trunk encapsulation dot1q
switchport mode trunk
negotiation auto
no cdp enable
channel-group 10 mode active
!
interface GigabitEthernet2/3
switchport trunk allowed vlan 526,700
switchport trunk encapsulation dot1q
switchport mode trunk
negotiation auto
no cdp enable
channel-group 10 mode active
!
interface Port-channel10
switchport trunk allowed vlan 526,700
switchport trunk encapsulation dot1q
switchport mode trunk
end

Nexus configuration (Both are identical with the keepalive IPs reversed)

vpc domain 10
peer-keepalive destination 10.10.1.12 source 10.10.1.11
!
interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 526,700
spanning-tree port type network
vpc peer-link
!
interface port-channel10
switchport mode trunk
switchport trunk allowed vlan 526,700
vpc 10
!
interface port-channel20
switchport mode trunk
switchport trunk allowed vlan 526
vpc 20
!
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan 526,700
channel-group 10 mode active
!
interface Ethernet1/2
switchport mode trunk
switchport trunk allowed vlan 526
channel-group 20 mode active
!
interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan 526,700
channel-group 1 mode active
!
interface Ethernet1/4
switchport mode trunk
switchport trunk allowed vlan 526,700
channel-group 1 mode active

Catalyst 2:

interface Port-channel20
switchport trunk allowed vlan 526
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface Vlan526
ip address 10.10.52.25 255.255.255.0
!
interface GigabitEthernet0/1
switchport trunk allowed vlan 526
switchport trunk encapsulation dot1q
switchport mode trunk
negotiation auto
channel-group 20 mode active
!
interface GigabitEthernet0/2
switchport trunk allowed vlan 526
switchport trunk encapsulation dot1q
switchport mode trunk
negotiation auto
channel-group 20 mode active

[balaji.bandi]

I have not seen any spanning tree config here. ( what version of nexus9k virtual you testing, i know some version do not work, even though it show up)

as per the config nexus only Layer 2 between these Switches ?

you can refer to one of my testings.

https://www.balajibandi.com/?s=VPC

[sanchezeldorado]

Hey Balaji,

Thank you for the reply. What Spanning tree configuration would be needed? I'm using cisco CML:

Software
BIOS: version
NXOS: version 9.2(4)
BIOS compile time:
NXOS image file is: bootflash:///nxos.9.2.4.bin
NXOS compile time: 8/20/2019 7:00:00 [08/20/2019 15:52:22]

Hardware
cisco Nexus9000 9000v Chassis

Here's the spanning tree from one of my catalyst switches on the port-channel 20 side.

VLAN0526
Spanning tree enabled protocol rstp
Root ID Priority 33294
Address 5254.0004.baf1
Cost 5
Port 65 (Port-channel20)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33294 (priority 32768 sys-id-ext 526)
Address 5254.000d.f871
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po20 Root FWD 4 128.65 P2p

Here's my spanning tree output and show vpc from the nexus that is up and running when the second peer is shut down. Note that the nexus switch is not passing traffic.

VLAN0526
Spanning tree enabled protocol rstp
Root ID Priority 33294
Address 5254.0004.baf1
Cost 1
Port 4105 (port-channel10)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33294 (priority 32768 sys-id-ext 526)
Address 5254.001a.ce84
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10 Root FWD 1 128.4105 (vPC) P2p
Po20 Desg FWD 1 128.4115 (vPC) P2p

NX11# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10
Peer status : peer link is down
vPC keep-alive status : Suspended (Destination IP not reachable)
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 2
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po1 down -

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
10 Po10 up success success 526,700



20 Po20 up success success 526

[balaji.bandi]

Looks your VPC link down (1 Po1 down - is this vPC Link) then the nexus become split brain here.

[sanchezeldorado]

The VPC link is down because I have the second vPC peer powered off completely. That's the problem. I would expect that the first vPC peer would continue passing traffic. If I have both vPC peers powered on, Po1 is up and everything works normally.

[balaji.bandi]

That is the Limitation of split brian of vPC. both become apart. remidation only to shutdown the one of the leg to work.

[sanchezeldorado]

I appreciate your help, but I don't think you're understanding what I'm saying. Correct me if I'm wrong, but split brain is where both switches are powered on, but both the peer link and the keepalive link are both disconnected. I do NOT have a split brain situation. One of my switches is completely shutdown and the one and only switch that is still online, will not pass traffic. As shown below, 10.10.1.2 cannot reach 10.10.1.25. Nexus 2 is completely shut down.

                                    Nexus 1

Catalyst1(10.10.1.2) <                    > Catalyst2(10.10.1.25)

                                    Nexus 2

-When I turn Nexus 2 back on, And Nexus 1 is also online, I am then able to reach 10.10.1.25.

-If I have nexus 1 shutdown, and nexus 2 online, I am still able to reach 10.10.1.25.

-My entire problem is when Nexus 2 is shutdown, I am not able to reach 10.10.1.25 whether Nexus 1 is online or not.