CISCO VPC:-primary/secondary switch suspends all VLANs

skempegowda · ‎05-09-2022

Hello

I am running VPC between 2 Cisco Nexus9k switches

When Primary switch is rebooted and comes up (port-channel VPC peer-link comes up) both primary and secondary switches suspend all VLANS

Switch1 (was the primary switch) that is being rebooted

2022 May 9 15:26:38 VALL-VALL-H-05-TORSW1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1000: Ethernet1/49 is up

<<< port-channel 1000 between switch1 and switch2 came up >>>>>

2022 May 9 15:26:38 VALL-VALL-H-05-TORSW1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1000: first operational port changed from none to Ethernet1/49

2022 May 9 15:26:38 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel1000,bandwidth changed to 100000000 Kbit

2022 May 9 15:26:38 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_UP: Interface Ethernet1/49 is up in mode trunk

<<< suspending all VLANS >>>>>>>>>

2022 May 9 15:26:38 VALL-VALL-H-05-TORSW1 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1-2,101-103,170-171,176,182-185,190,201-203,210,220,230,240,1000-1002,1170-1171,1176,1182-1185,1190,1

470-1471,1476,1482-1485,1490,1570-1571,1576,1582-1585,1590,1670-1671,1676,1682-1685,1690,1770-1771,1776,1782-1785,1790,2000-2002 on Interf

2022 May 9 15:26:38 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_UP: Interface port-channel1000 is up in mode trunk

2022 May 9 15:26:39 VALL-VALL-H-05-TORSW1 %ETHPORT-5-SPEED: Interface Ethernet1/43, operational speed changed to 1 Gbps

2022 May 9 15:26:39 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/43, operational duplex mode changed to Full

2022 May 9 15:26:39 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/43, operational Receive Flow Control state changed to off

2022 May 9 15:26:39 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/43, operational Transmit Flow Control state changed to off

2022 May 9 15:26:39 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_UP: Interface Ethernet1/43 is up in mode trunk

2022 May 9 15:26:40 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_DOWN_NONE: Interface port-channel160 is down (None)

2022 May 9 15:26:40 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_DOWN_NONE: Interface port-channel159 is down (None)

2022 May 9 15:26:40 VALL-VALL-H-05-TORSW1 %ETHPORT-5-IF_DOWN_NONE: Interface port-channel158 is down (None)

Switch2 (that is up and running) while switch1 is being rebooted

==============================================

2022 May 9 15:28:23 VALL-VALL-H-06-TORSW2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel1000,bandwidth changed to 100000000 Kbit

2022 May 9 15:28:23 VALL-VALL-H-06-TORSW2 %ETHPORT-5-IF_UP: Interface Ethernet1/49 is up in mode trunk

2022 May 9 15:28:23 VALL-VALL-H-06-TORSW2 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1-2,101-103,170-171,176,182-185,190,201-203,210,220,230,240,1000-1002,1170-1171,1176,1182-1185,1190,1

470-1471,1476,1482-1485,1490,1570-1571,1576,1582-1585,1590,1670-1671,1676,1682-1685,1690,1770-1771,1776,1782-1785,1790,2000-2002 on Interf

2022 May 9 15:28:23 VALL-VALL-H-06-TORSW2 %ETHPORT-5-IF_UP: Interface port-channel1000 is up in mode trunk

2022 May 9 15:28:25 VALL-VALL-H-06-TORSW2 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1-2,101-103,170-171,176,182-185,190,201-203,210,220,230,240,1000-1002,1170-1171,1176,1182-1185,1190,147

0-1471,1476,1482-1485,1490,1570-1571,1576,1582-1585,1590,1670-1671,1676,1682-1685,1690,1770-1771,1776,1782-1785,1790,2000-2002 on Interfac

MHM Cisco World · ‎05-09-2022

share output of
show VPC
show VPC brief

marce1000 · ‎05-10-2022

- Also post output of : show logging logfile | include ignore-case suspend

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

skempegowda · ‎05-10-2022

steady state :-

===================
VALL-VALL-H-05-TORSW1#
VALL-VALL-H-05-TORSW1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 100
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary, operational secondary
Number of vPCs configured : 17
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Enabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po1000 up 1-2,101-103,170-171,176,182-185,190,201-203,210,
220,230,240,1000-1002,1170-1171,1176,1182-1185,
1190,1470-1471,1476,1482-1485,1490,1570-1571,1576
,1582-1585,1590,1670-1671,1676,1682-1685,1690,
1770-1771,1776,1782-1785,1790,2000-2002

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
132 Po132 up success success 2

133 Po133 up success success 2

134 Po134 up success success 2

135 Po135 up success success 2

136 Po136 up success success 2

137 Po137 up success success 201-203

140 Po140 up success success 101-103

141 Po141 up success success 201-203

142 Po142 up success success 101-103

146 Po146 up success success 170-171,176,
1170-1171,1176,
1470-1471,1476,
1570-1571,1576,
1670-1671,1676,
1770-1771,1776
148 Po148 up success success 182-185,190,
1182-1185,1190,
1482-1485,1490,
1582-1585,1590,
1682-1685,1690,
1782-1785,1790
155 Po155 up success success 201-203,1570-1571,
1576,1582-1585,
1590
156 Po156 up success success 101-103,1670-1671,
1676,1682-1685,
1690
157 Po157 up success success 201-203,1770-1771,
1776,1782-1785,
1790
158 Po158 up success success 101-103,170-171,
176,182-185,190
159 Po159 up success success 201-203,1170-1171,
1176,1182-1185,
1190
160 Po160 up success success 101-103,1470-1471,
1476,1482-1485,
1490

VALL-VALL-H-05-TORSW1# show vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (594) seconds, (652) msec
--Send status : Success
--Last send at : 2022.05.06 14:13:57 597 ms
--Sent on interface : mgmt0
--Receive status : Success
--Last receive at : 2022.05.06 14:13:57 781 ms
--Received on interface : mgmt0
--Last update from peer : (0) seconds, (121) msec

vPC Keep-alive parameters
--Destination : 192.168.2.101
--Keepalive interval : 400 msec
--Keepalive timeout : 3 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : management
--Keepalive udp port : 3200
--Keepalive tos : 192
VALL-VALL-H-05-TORSW1# show vp role

vPC Role status
----------------------------------------------------
vPC role : primary, operational secondary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:64
vPC system-priority : 100
vPC local system-mac : d0:e0:42:64:e2:d7
vPC local role-priority : 10
vPC local config role-priority : 10
vPC peer system-mac : d0:e0:42:64:e3:1f
vPC peer role-priority : 20
vPC peer config role-priority : 20
VALL-VALL-H-05-TORSW1#
VALL-VALL-H-05-TORSW1#
VALL-VALL-H-05-TORSW1#

skempegowda · ‎05-10-2022

MHM Cisco World · ‎05-10-2022

peer-keepalive destination 192.168.2.100 source 192.168.2.101

If I am right
this appear in both Nexus ???
It must mirror i.e.
peer-keepalive destination will source in other peer.

paul driver · ‎05-10-2022

Hello
The primary vpc switch reboots the secondary takes over and stays primary even when original primary comes back

If the vpc peer link is down but the keep-alive isn’t then as the secondary vpc switch WILL suspend all vpc member links and svi vlans

if both links are down then you have isolated vpc switches which cannot sync with each other as such the secondary is suspended

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

skempegowda · ‎05-11-2022

Thank you

I had to enable below command under VPC VLAN port-channels , when ToR1 (primary) was rebooted this time ToR2 did not suspend VLANs (in LACP fast mode)

no lacp graceful-convergence
no lacp suspend-individual

Regards
Shwetha

MHM Cisco World · ‎05-11-2022

No this is no solution here is not permanent one,
the Keepalive link is misconfig in one Peer and with auto-recovery is lead to this case.
YOU NEED TO RIGHT CONFIG THE KEEPALIVE LINK.

skempegowda · ‎05-11-2022

Hi,

Thank you for your response

Below is the config , could you help what is the misconfig ?

ToR1 and ToR2 reachable via mgmt. interfaces

ToR1:-
interface mgmt0
speed 100
vrf member management
ip address 192.168.2.100/24

vpc domain 100
role priority 10
system-priority 100
peer-keepalive destination 192.168.2.101 interval 400 timeout 3
delay restore 15
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize

ToR2:-

interface mgmt0
speed 100
vrf member management
ip address 192.168.2.101/24

vpc domain 100
role priority 20
system-priority 100
peer-keepalive destination 192.168.2.100 interval 400 timeout 3
delay restore 15
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize

MHM Cisco World · ‎05-11-2022

I depend on this EXCEL,
it TOR1 vs TOR2
peer-keepalive destination is same ? this make me think that
this misconfig and need to reconfig again right.

skempegowda · ‎05-12-2022

Hi,

Thanks for your help
There is no misconfig on keepalive. Here is the config as posted before
No lacp suspend-individual helped in our case (Cisco Tech support guy from Ireland suggested this solution)

interface port-channel137
description LAG-vCCAP2 CP1
switchport
switchport mode trunk
switchport trunk allowed vlan 201-203
spanning-tree port type edge trunk
mtu 9000
no lacp graceful-convergence >>> Below config worked for us >>>>>
no lacp suspend-individual

ToR1 and ToR2 reachable via mgmt. interfaces (No misconfig)

ToR1:-
interface mgmt0
speed 100
vrf member management
ip address 192.168.2.100/24

vpc domain 100
role priority 10
system-priority 100
peer-keepalive destination 192.168.2.101 interval 400 timeout 3
delay restore 15
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize

ToR2:-

interface mgmt0
speed 100
vrf member management
ip address 192.168.2.101/24

vpc domain 100
role priority 20
system-priority 100
peer-keepalive destination 192.168.2.100 interval 400 timeout 3
delay restore 15
peer-gateway
auto-recovery
ipv6 nd synchronize
ip arp synchronize