Solved: VPC on Nexus 9k have not come up after an upgrade

PacketSpartan · ‎04-05-2022

We have 2 Nexus 9380s (.7.0.3.I6.1) and we upgraded the primary one to nxos.7.0.3.I7.10, Some of the VPCs have not come back up, under show vpc we are seeing Consistency Check Not - Performed

Looking in the logs, we're not seeing anything that pops out, could it be the fact that our 9ks are on different versions?

# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary, operational secondary
Number of vPCs configured : 12
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po100 up 1,5,3940-3941

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
22 Po22 down* Not Consistency Check Not -
Applicable Performed
23 Po23 down* Not Consistency Check Not -
Applicable Performed
32 Po32 down* Not Consistency Check Not -
Applicable Performed
33 Po33 down* Not Consistency Check Not -
Applicable Performed
44 Po44 down* Not Consistency Check Not -
Applicable Performed
45 Po45 down* Not Consistency Check Not -
Applicable Performed
54 Po54 down* Not Consistency Check Not -
Applicable Performed
55 Po55 down* Not Consistency Check Not -
Applicable Performed

CCNA R&S

Sergiu.Daniluk · ‎04-07-2022

If you set the flow control off on all ports on both sides, does the PO comes up?

Sergiu

View solution in original post

Sergiu.Daniluk · ‎04-05-2022

Hi @PacketSpartan

Consistency check is performed once the interface comes up.

You should check the status of the interfaces and of the protchannel.

Can you share:

show port-channel summary
show interface status
show interface port-channel 22

Also, share the peer device:

show vpc
show port-channel summary

Cheers,

Sergiu

PacketSpartan · ‎04-06-2022

Hi Sergiu

Thank you for the reply,

# sh port-channel summary 
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
22 Po22(SD) Eth LACP Eth1/1(s) Eth1/2(s) Eth1/3(s)
23 Po23(SD) Eth LACP Eth1/15(D) 
32 Po32(SD) Eth LACP Eth1/39(D) 
33 Po33(SD) Eth LACP Eth1/25(s) Eth1/26(s) Eth1/27(s)
44 Po44(SD) Eth LACP Eth1/4(s) Eth1/5(s) Eth1/6(s)
45 Po45(SD) Eth LACP Eth1/14(D) 
48 Po48(RU) Eth LACP Eth1/47(P) Eth1/48(P) 
54 Po54(SD) Eth LACP Eth1/38(D) 
55 Po55(SD) Eth LACP Eth1/28(s) Eth1/29(s) Eth1/30(s)
88 Po88(SD) Eth LACP Eth1/10(D) Eth1/11(D) Eth1/12(D)
89 Po89(SU) Eth LACP Eth1/13(P) 
98 Po98(SU) Eth LACP Eth1/37(P) 
99 Po99(SD) Eth LACP Eth1/34(D) Eth1/35(D) Eth1/36(D)
100 Po100(SU) Eth LACP Eth1/52(P) Eth1/54(P)

# sh interface status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
mgmt0 -- connected routed full 1000 --

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 xxxxx-D1 suspended 3940 auto auto 1000base-LH
Eth1/2 xxxxx-D1 suspended 3940 auto auto 1000base-LH
Eth1/3xxxxx-D1 suspended 3940 auto auto 1000base-LH

# sh interface port-channel 22
port-channel22 is down (No operational members)
admin state is up,
vPC Status: Down, vPC number: 22 [packets forwarded via vPC peer-link]
Hardware: Port-Channel, address: 4c77.6d9e.fe2d (bia 4c77.6d9e.fe2d)
Description: xxxxxx-D1 Po21
MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
auto-duplex, auto-speed
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off 
EtherType is 0x8100 
Members in this channel: Eth1/1, Eth1/2, Eth1/3
Last clearing of "show interface" counters never
0 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 80 bits/sec, 0 packets/sec
30 seconds output rate 480 bits/sec, 0 packets/sec
input rate 80 bps, 0 pps; output rate 480 bps, 0 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 216 bits/sec, 0 packets/sec
300 seconds output rate 720 bits/sec, 0 packets/sec
input rate 216 bps, 0 pps; output rate 720 bps, 0 pps
RX
0 unicast packets 25420 multicast packets 0 broadcast packets
25420 input packets 3828280 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 57228 multicast packets 0 broadcast packets
57228 output packets 11784044 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

PEER switch

C2# sh vpc 
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10 
Peer status : peer adjacency formed ok 
vPC keep-alive status : peer is alive 
Configuration consistency status : success 
Per-vlan consistency status : success 
Type-2 consistency status : success 
vPC role : secondary, operational primary
Number of vPCs configured : 12 
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans 
-- ---- ------ -------------------------------------------------
1 Po100 up 1,5,3940-3941

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
22 Po22 up success success 3940 

23 Po23 down* success success - 

32 Po32 down* success success - 

33 Po33 up success success 3940 

44 Po44 up success success 3940 

45 Po45 down* success success - 

54 Po54 down* success success - 

55 Po55 up success success 3940 

88 Po88 down* success success - 

89 Po89 up success success 3940 

98 Po98 up success success 3940 

99 Po99 down* success success -

Please check "show vpc consistency-parameters vpc <vpc-num>" for the 
consistency reason of down vpc and for type-2 consistency reasons for 
any vpc.

C2#

C2# sh port-channel summary 
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SD) Eth NONE --
22 Po22(SU) Eth LACP Eth1/1(P) Eth1/2(P) Eth1/3(P)
23 Po23(SD) Eth LACP Eth1/15(D) 
32 Po32(SD) Eth LACP Eth1/39(D) 
33 Po33(SU) Eth LACP Eth1/25(P) Eth1/26(P) Eth1/27(P)
44 Po44(SU) Eth LACP Eth1/4(P) Eth1/5(P) Eth1/6(P)
45 Po45(SD) Eth LACP Eth1/14(D) 
48 Po48(RU) Eth LACP Eth1/47(P) Eth1/48(P) 
54 Po54(SD) Eth LACP Eth1/38(D) 
55 Po55(SU) Eth LACP Eth1/28(P) Eth1/29(P) Eth1/30(P)
88 Po88(SD) Eth LACP Eth1/10(D) Eth1/11(D) Eth1/12(D)
89 Po89(SU) Eth LACP Eth1/13(P) 
98 Po98(SU) Eth LACP Eth1/37(P) 
99 Po99(SD) Eth LACP Eth1/34(D) Eth1/35(D) Eth1/36(D)
100 Po100(SU) Eth LACP Eth1/52(P) Eth1/54(P) 
C2#

The Port channel is between our distribution switch and the Nexus C1, (one to C2 is up). it was C1 that was upgraded

- we have bounced the Portchannel on the C1

- We have reconfigured the port channel members on the Distribution switch

- still ends up being suspended

Here is the log from Distribution switch

Apr 6 08:36:33.708 BST: %EC-5-CANNOT_BUNDLE2: Gi6/46 is not compatible with Gi7/46 and will be suspended (Operational flow control send of Gi6/46 is off, Gi7/46 is on)
Apr 6 08:36:33.708 BST: %EC-5-CANNOT_BUNDLE2: Gi6/47 is not compatible with Gi7/46 and will be suspended (Operational flow control send of Gi6/47 is off, Gi7/46 is on)
Apr 6 08:36:33.708 BST: %EC-5-CANNOT_BUNDLE2: Gi6/48 is not compatible with Gi7/46 and will be suspended (Operational flow control send of Gi6/48 is off, Gi7/46 is on)
Apr 6 08:36:33.708 BST: %EC-5-CANNOT_BUNDLE2: Gi6/46 is not compatible with Gi7/46 and will be suspended (Operational flow control send of Gi6/46 is off, Gi7/46 is on)
Apr 6 08:36:33.708 BST: %EC-5-CANNOT_BUNDLE2: Gi6/47 is not compatible with Gi7/46 and will be suspended (Operational flow control send of Gi6/47 is off, Gi7/46 is on)
Apr 6 08:36:33.708 BST: %EC-5-CANNOT_BUNDLE2: Gi6/48 is not compatible with Gi7/46 and will be suspended (Operational flow control send of Gi6/48 is off, Gi7/46 is on)
Apr 6 08:36:33.720 BST: %LINK-3-UPDOWN: Interface GigabitEthernet6/46, changed state to up
Apr 6 08:36:33.720 BST: %LINK-3-UPDOWN: Interface GigabitEthernet6/47, changed state to up
Apr 6 08:36:33.720 BST: %LINK-3-UPDOWN: Interface GigabitEthernet6/48, changed state to up

From the nexus 9k C1, when we bounced the port channel

2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface port-channel22 is down(Config change)
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface port-channel22 is down (Administratively down)
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/2 is down(Config change)
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/3 is down(Config change)
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/1 is down(Config change)
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/2 is down (Channel admin down)
2022 Apr 6 08:51:40 GLA1064-C1 last message repeated 1 time
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/3 is down (Channel admin down)
2022 Apr 6 08:51:40 GLA1064-C1 last message repeated 1 time
2022 Apr 6 08:51:40 GLA1064-C1 %ETHPORT-5-IF_DOWN_CHANNEL_ADMIN_DOWN: Interface Ethernet1/1 is down (Channel admin down)
2022 Apr 6 08:51:46 GLA1064-C1 last message repeated 1 time
2022 Apr 6 08:51:46 GLA1064-C1 %ETHPORT-5-IF_ADMIN_UP: Interface port-channel22 is admin up .
2022 Apr 6 08:51:46 GLA1064-C1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel22 is down (No operational members)
2022 Apr 6 08:51:47 GLA1064-C1 last message repeated 1 time
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-SPEED: Interface Ethernet1/2, operational speed changed to 1 Gbps
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/2, operational duplex mode changed to Full
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/2, operational Receive Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/2, operational Transmit Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-SPEED: Interface Ethernet1/3, operational speed changed to 1 Gbps
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/3, operational duplex mode changed to Full
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/3, operational Receive Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/3, operational Transmit Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-SPEED: Interface Ethernet1/1, operational speed changed to 1 Gbps
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/1, operational duplex mode changed to Full
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/1, operational Receive Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/1, operational Transmit Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-SPEED: Interface port-channel22, operational speed changed to 1 Gbps
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_DUPLEX: Interface port-channel22, operational duplex mode changed to Full
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel22, operational Receive Flow Control state changed to off
2022 Apr 6 08:51:47 GLA1064-C1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel22, operational Transmit Flow Control state changed to off
2022 Apr 6 08:51:57 GLA1064-C1 %LACP-5-LACP_SUSPEND_INDIVIDUAL: LACP port Ethernet1/2(0x1a000200) of port-channel Ethernet1/2(0x1a000200) not receiving any LACP BPDUs suspending (individual) port 
2022 Apr 6 08:51:57 GLA1064-C1 %LACP-5-LACP_SUSPEND_INDIVIDUAL: LACP port Ethernet1/3(0x1a000400) of port-channel Ethernet1/3(0x1a000400) not receiving any LACP BPDUs suspending (individual) port 
2022 Apr 6 08:51:57 GLA1064-C1 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/2: Ethernet1/2 is suspended
2022 Apr 6 08:51:57 GLA1064-C1 %LACP-5-LACP_SUSPEND_INDIVIDUAL: LACP port Ethernet1/1(0x1a000000) of port-channel Ethernet1/1(0x1a000000) not receiving any LACP BPDUs suspending (individual) port 
2022 Apr 6 08:51:57 GLA1064-C1 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/3: Ethernet1/3 is suspended
2022 Apr 6 08:51:57 GLA1064-C1 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/1: Ethernet1/1 is suspended

CCNA R&S

Sergiu.Daniluk · ‎04-06-2022

Hi @PacketSpartan

To me, it looks like the problem is on Dist switch, and is related to flow control.

You should disable or enable flow control on all ports, on both Nexus and or Cat, depending on the needs.

Stay safe,

Sergiu

PacketSpartan · ‎04-06-2022

Hi Sergiu

We've tried to configure the Flow control on the 6807' ports that are set to off, but unable to match the other ports in the port channel.

Which ever combination of commands we use, we cant set it to desired and on

From# sh etherchannel 21 det

Probable reason: Operational flow control send of Gi6/46 is off, Gi7/46 is on

Show flowcontrol output

Gi6/46 desired off off off 0 0 - To C1

Gi6/47 desired off off off 0 0 - To C1

Gi6/48 desired off off off 0 0 - To C1

Gi7/46 desired on off off 0 0 - To C2

Gi7/47 desired on off off 0 0 - To C2

Gi7/48 desired on off off 0 0 - To C2

CCNA R&S

Sergiu.Daniluk · ‎04-07-2022

If you set the flow control off on all ports on both sides, does the PO comes up?

Sergiu

PacketSpartan · ‎04-07-2022

Hi Sergiu

Thank you for the reply, We ended up raising a TAC case and this is what we did.

We had to change the Flow control to "Send off" on the downstream cat 6807 switch to match the port channel members that were not forming the port channel. The nexus was already at off by default (existing config)

Once we changed it on all the live ports, from desired on to Send off, The port channel was fully back up.

Thanks a lot for your help again.

CCNA R&S