Nexus 9000 VLANs suspended when primary forms adjacency

[Simon2000]

I have a pair of Nexus 9000 switches configured with a vpc peer link using eth1/53 and eth1/54 (configured as port channel 128)

I have two Window hosts connected using LACP VPCs across the two switches, with a continual ping going from host 1 to host 2

If I reload the switch acting as the vpc primary then the ping traffic continues and the switch acting as the vpc secondary becomes the operational primary.

However when the switch that was acting as vpc primary starts forming the adjacency again then I see pings being dropped for 8-10 seconds.

In the log on the acting primary I see the following:-

%ETHPORT-5-SPEED: Interface Ethernet1/53, operational speed changed to 100 Gbps
%ETHPORT-5-IF_DUPLEX: Interface Ethernet1/53, operational duplex mode changed to Full
%ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/53, operational Receive Flow Control state changed to off
%ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/53, operational Transmit Flow Control state changed to off
%ETHPORT-5-SPEED: Interface port-channel128, operational speed changed to 100 Gbps
%ETHPORT-5-IF_DUPLEX: Interface port-channel128, operational duplex mode changed to Full
%ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel128, operational Receive Flow Control state changed to off
%ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel128, operational Transmit Flow Control state changed to off
%ETHPORT-5-SPEED: Interface Ethernet1/54, operational speed changed to 100 Gbps
%ETHPORT-5-IF_DUPLEX: Interface Ethernet1/54, operational duplex mode changed to Full
%ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/54, operational Receive Flow Control state changed to off
%ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/54, operational Transmit Flow Control state changed to off
%ETH_PORT_CHANNEL-5-PORT_UP: port-channel128: Ethernet1/53 is up
%ETH_PORT_CHANNEL-5-PORT_UP: port-channel128: Ethernet1/54 is up
%ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel128: first operational port changed from none to Ethernet1/53
%ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel128,bandwidth changed to 100000000 Kbit
%ETHPORT-5-IF_UP: Interface Ethernet1/53 is up in mode trunk

%ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 31,35,40-42

%ETHPORT-5-IF_UP: Interface port-channel128 is up in mode trunk
%ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel128,bandwidth changed to 200000000 Kbit
%ETHPORT-5-IF_UP: Interface Ethernet1/54 is up in mode trunk

%ETHPORT-3-IF_ERROR_VLANS_REMOVED:VLANs 31,35,40-42

The dropped pings start at the "IF_ERROR_VLANS_SUSPENDED" message and resume at the "IF_ERROR_VLANS_REMOVED" message

Is this expected behaviour? It is as if the peer link config between the two switches is mismatched until Ethernet1/54 is trunking.

[MHM Cisco World]

Show vpc breif 

Show etherchannel summary 

Share this when issue appear

MHM

[Simon2000]

As requested:-

Switch1# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 14
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 4
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 60s)
Delay-restore SVI status : Timer is off.(timeout = 20s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po128 up 31,35,40-42,

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
1 Po1 up success success 40-42

2 Po2 up success success 40-42

3 Po3 down* success success -

50 Po120 down* success success -

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

Switch1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/1(P)
2 Po2(SU) Eth LACP Eth1/2(P)
3 Po3(SD) Eth LACP Eth1/3(D)
120 Po120(SD) Eth LACP Eth1/45(D) Eth1/46(D) Eth1/47(D)
Eth1/48(D)
128 Po128(SU) Eth LACP Eth1/53(P) Eth1/54(P)

[MHM Cisco World]

Which PO have issue ?

MHM

[Simon2000]

Apologies, this is the output from the two commands during the issue:-

Switch1# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 14
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary, operational secondary
Number of vPCs configured : 4
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 60s)
Delay-restore SVI status : Timer is on.(timeout = 20s, 13s left)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po128 up 31,35,40-42

Switch1# show port-channel summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SD) Eth LACP Eth1/1(D)
2 Po2(SD) Eth LACP Eth1/2(D)
3 Po3(SD) Eth LACP Eth1/3(D)
120 Po120(SD) Eth LACP Eth1/45(D) Eth1/46(D) Eth1/47(D)
Eth1/48(D)
128 Po128(SU) Eth LACP Eth1/53(P) Eth1/54(P)

Switch2# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 14
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary, operational primary
Number of vPCs configured : 4
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 60s)
Delay-restore SVI status : Timer is on.(timeout = 20s, 12s left)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po128 up 31,35,40-42

Switch2# show port-channel summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/1(P)
2 Po2(SU) Eth LACP Eth1/2(P)
3 Po3(SD) Eth LACP Eth1/3(D)
120 Po120(SD) Eth LACP Eth1/45(D) Eth1/46(D) Eth1/47(D)
Eth1/48(D)
128 Po128(SU) Eth LACP Eth1/53(P) Eth1/54(P)

All port channels are affected. The test ping is between port channel 1 and port channel 2 but is the same for all port channels 

[MHM Cisco World]

Are you use dedicated l3 interface for keepalive or you use svi of  vlan over peer link trunk?

[MHM Cisco World]

1-I think you miss add PO in show vpc brief 
can you check it appear or not ?

2- I ask if you config keep alive as dedicate or with peer link 
3- can I see show spanning tree 
4- show lacp counter 
5- show lacp neighbor 

are you sure windows support LACP, I think you need to change Mode to ON
but let check this first 

MHM

[Simon2000]

1. Here is the full show vpc brief

Switch2# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 14
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary, operational primary
Number of vPCs configured : 4
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 60s)
Delay-restore SVI status : Timer is on.(timeout = 20s, 17s left)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po128 up 31,35,40-42

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
1 Po1 up success success 40-42,46-48
2 Po2 up success success 40-42,46-48
3 Po3 down* success success -
50 Po120 down* success success -

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

2. The keep alive is running over the MGMT interfaces. Currently its a direct connect as its in test but I have tested using an Ethernet switch with the same results.

3. Output from show spanning tree

Switch2# show span

VLAN0031
Spanning tree enabled protocol rstp
Root ID Priority 32799
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32799 (priority 32768 sys-id-ext 31)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0035
Spanning tree enabled protocol rstp
Root ID Priority 32803
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32803 (priority 32768 sys-id-ext 35)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0040
Spanning tree enabled protocol rstp
Root ID Priority 32808
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32808 (priority 32768 sys-id-ext 40)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg LRN 1 128.4096 (vPC) P2p
Po2 Desg LRN 1 128.4097 (vPC) P2p
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0041
Spanning tree enabled protocol rstp
Root ID Priority 32809
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32809 (priority 32768 sys-id-ext 41)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg LRN 1 128.4096 (vPC) P2p
Po2 Desg LRN 1 128.4097 (vPC) P2p
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0042
Spanning tree enabled protocol rstp
Root ID Priority 32810
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32810 (priority 32768 sys-id-ext 42)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg LRN 1 128.4096 (vPC) P2p
Po2 Desg LRN 1 128.4097 (vPC) P2p
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0046
Spanning tree enabled protocol rstp
Root ID Priority 32814
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32814 (priority 32768 sys-id-ext 46)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg LRN 1 128.4096 (vPC) P2p
Po2 Desg LRN 1 128.4097 (vPC) P2p
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0047
Spanning tree enabled protocol rstp
Root ID Priority 32815
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32815 (priority 32768 sys-id-ext 47)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg LRN 1 128.4096 (vPC) P2p
Po2 Desg LRN 1 128.4097 (vPC) P2p
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

VLAN0048
Spanning tree enabled protocol rstp
Root ID Priority 32816
Address 003a.9c4b.54a7
Cost 1
Port 4223 (port-channel128)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32816 (priority 32768 sys-id-ext 48)
Address 003a.9c4d.f7a7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg LRN 1 128.4096 (vPC) P2p
Po2 Desg LRN 1 128.4097 (vPC) P2p
Po128 Root FWD 1 128.4223 (vPC peer-link) Network P2p

4 Show lacp counter

Switch2# show lacp counters
NOTE: Clear lacp counters to get accurate statistics

------------------------------------------------------------------------------
LACPDUs Markers/Resp LACPDUs
Port Sent Recv Recv Sent Pkts Err
------------------------------------------------------------------------------
port-channel1
Ethernet1/1 1259 53 0 0 0

port-channel2
Ethernet1/2 1328 55 0 0 0

port-channel128
Ethernet1/53 60 57 0 0 0

Ethernet1/54 57 57 0 0 0

5 show lacp neighbor

Switch2# show lacp nei
Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
port-channel1 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/1 0,0-90-fa-96-47-5e 0x2 2982 FA

LACP Partner Partner Partner
Port Priority Oper Key Port State
0 0x0 0x3f

Po2 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/2 0,0-90-fa-c8-6-1e 0x2 2981 FA

LACP Partner Partner Partner
Port Priority Oper Key Port State
0 0x0 0x3f

Po128 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/53 32768,0-3a-9c-4b-54-a7 0x1d1 183 SA

LACP Partner Partner Partner
Port Priority Oper Key Port State
32768 0x7f 0x3d

Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/54 32768,0-3a-9c-4b-54-a7 0x1d5 183 SA

LACP Partner Partner Partner
Port Priority Oper Key Port State
32768 0x7f 0x3d

[MHM Cisco World]

Get it' 

Vlan 46-48 is not allow in peer link trunk 

So it break vpc

Why you not include it?

MHM

[Simon2000]

Apologies they are allowed but got removed when I scraped the output.

Switch2# sho vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 14
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary, operational primary
Number of vPCs configured : 4
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 60s)
Delay-restore SVI status : Timer is off.(timeout = 20s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po128 up 31,35,40-42,46-48

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
1 Po1 up success success 40-42,46-48
2 Po2 up success success 40-42,46-48
3 Po3 down* success success -
50 Po120 down* success success -

Please check "show vpc consistency-parameters vpc <vpc-num>" for the
consistency reason of down vpc and for type-2 consistency reasons for
any vpc.

[MHM Cisco World]

Hi Friend 
sorry I make you waiting but I run lab to follow with you step need for troubleshooting 
when issue appear in both nexus share (if you can add it to text file it more easy)

switch# show vpc consistency-parameters vpc 1/2/3/120/128 <<- one by one 
switch# show vpc consistency-parameters global
switch# show vpc consistency-parameters vlans

waiting your replay 
Thanks alot

MHM

[Simon2000]

Apologies for the delay. 

I've run the commands during the issue and attached the results from the two switches

[MHM Cisco World]

thanks for more detail 
the two doc. you share second one is good the first one the table is not past correctly 
I run lab to follow with you 
the show vpc consi vpc 1/2 dont show any thing from the peer only from local 
are you sure you run PO with vpc 1 and 2 in both NSK 

MHM

[MHM Cisco World]

also 
show vpc role <<- in both NSK not work as primary 

MHM

[Simon2000]

I think this is because the reloaded switch is still forming a proper adjacency. If I run the command now on the reloaded switch it looks fine.