Nexus 31108PC-V and Nexus 31108TC-V dual sided VPC issue

[Caranrack]

Hi!

I would like to ask for help due to issue that I am currently experiencing while configuring our pairs of Nexus 31108PC-V and Nexus 31108TC-V in dual sided VPC.

Afer configuring basic prerequisites for VPC function, one of my switches in second pair got left out of VPC. I tried everything, but cannot get port-channel on that switch to be part of VPC.

Only clue that I get in logs are following messages:

2022 May 16 18:15:36 N3K-DC-1 %ETHPORT-5-IF_DUPLEX: Interface port-channel3, ope
rational duplex mode changed to Full
2022 May 16 18:15:36 N3K-DC-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-chan
nel3, operational Receive Flow Control state changed to off
2022 May 16 18:15:36 N3K-DC-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-chan
nel3, operational Transmit Flow Control state changed to off

VPC status on all my switches is the following:

N3K-AGG-1# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po2 up 1,5,13,43,60,88,90,100-110,120-123,140,150,

169-172,192,200-209,219-220

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
3 Po3 up success success 1,5,13,43,60,88,90

,100-110,120-123,

140,150,169-172,

192,200-209,

219-220

N3K-AGG-2# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 1
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po2 up 1,5,13,43,60,88,90,100-110,120-123,140,150,

169-172,192,200-209,219-220

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
3 Po3 up success success 1,5,13,43,60,88,90

,100-110,120-123,

140,150,169-172,

192,200-209,

219-220

N3K-DC-1# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 2
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 1
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po2 up 1,5,13,43,60,88,90,100-110,120-123,140,150,

169-172,192,200-209,219-220

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
3 Po3 down* success success -

N3K-DC-2# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 2
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po2 up 1,5,13,43,60,88,90,100-110,120-123,140,150,

169-172,192,200-209,219-220

vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
3 Po3 up success success 1,5,13,43,60,88,90

,100-110,120-123,

140,150,169-172,

192,200-209,

219-220

I have attached my configuration and I do hope that someone is willing to take a look and enlighten me what did I do wrong.

[balaji.bandi]

Do you have a small network diagram of what port is connected where   ? are you trying for enhanced vPC ? or a Straight port channel?

Mean time let me review the config and get back to you what my findings ?

what Nexus code running on both the pairs ?

[Caranrack]

Thank you for your answer.

I will draw diagram and post it here.

This is straight port channel, not enchanced vpc.

All switches are upgraded to latest nxos 9.3.9.

Sure, feel free to check the config. Thank you for that.

[Caranrack]

I created diagram with connections. Hope you can see how it is configured.

[balaji.bandi]

how some not able to view your clear diagram, here my testing for reference :

https://www.balajibandi.com/?m=201802

[Caranrack]

Sorry, I don't see anything usefull there. Can you point me to anything that might help?

[balaji.bandi]

Only usefull where you look at  and I was referring diagram, the one posted i can not view it( are you ?) from the post, what port connected where) ..so post digram which can match your config.

[Caranrack]

But i allready posted the diagram. Here it is, once more:

[MHM Cisco World]

I see many case 
the Engineer misconfig the 
domain in upper NSK peer and lower NSK peer must different 
the vPC use for connect all "four" NSK but be same.
please refer to photo above 

[Caranrack]

Thank you for your answer.

The diagram that you provided looks like my configuration. I used different domains for pairs and same VPC and PO name for cross-link.

But, does PO naming for peer link needs to be different? Since mine is the same on both pairs.

[MHM Cisco World]

NO issue with Keepalive and Peer-Link it OK even the PO member is OK.
but for checking can you ping through keep alive i.e. ping NSK-2 source NSK-1??

One Last Point, I never See config PO for keep alive so can you config one giga port "no switchport " instead of config PO.

second 

AGG<-->DC_L2_link << this discerption in all PO vPC between two Pair,
can you check the cabling ?
there are four port in each NSK 
each two must connect to different NSK peer in other Pair.
just check thus point.

[Caranrack]

Here are my connections.

 

[Caranrack]

Just out of pure despair I changed config on DC pair to use MGMT as keepalive. Result is the same.

I triple checked all my cabling and it is all good. 

Anything more that I can try?

[MHM Cisco World]

show vpc peer-keepalive <- check the status of keep alive for all four NSK 
 please share output here 

[Caranrack]

Sure, no problem:

N3K-AGG-1# show vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (83877) seconds, (764) msec
--Send status : Success
--Last send at : 2022.05.17 13:10:27 730 ms
--Sent on interface : Po1
--Receive status : Success
--Last receive at : 2022.05.17 13:10:27 731 ms
--Received on interface : Po1
--Last update from peer : (0) seconds, (256) msec

vPC Keep-alive parameters
--Destination : 172.16.12.255
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : VPCkeepalive
--Keepalive udp port : 3200
--Keepalive tos : 192

N3K-AGG-2# show vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (83881) seconds, (910) msec
--Send status : Success
--Last send at : 2022.05.17 13:12:00 918 ms
--Sent on interface : Po1
--Receive status : Success
--Last receive at : 2022.05.17 13:12:00 919 ms
--Received on interface : Po1
--Last update from peer : (0) seconds, (52) msec

vPC Keep-alive parameters
--Destination : 172.16.12.254
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : VPCkeepalive
--Keepalive udp port : 3200
--Keepalive tos : 192

N3K-DC-1# show vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (1943) seconds, (444) msec
--Send status : Success
--Last send at : 2022.05.17 13:05:00 748 ms
--Sent on interface : mgmt0
--Receive status : Success
--Last receive at : 2022.05.17 13:05:00 288 ms
--Received on interface : mgmt0
--Last update from peer : (0) seconds, (694) msec

vPC Keep-alive parameters
--Destination : 172.16.34.254
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : management
--Keepalive udp port : 3200
--Keepalive tos : 192

N3K-DC-2# show vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (1947) seconds, (500) msec
--Send status : Success
--Last send at : 2022.05.17 13:05:05 298 ms
--Sent on interface : mgmt0
--Receive status : Success
--Last receive at : 2022.05.17 13:05:04 758 ms
--Received on interface : mgmt0
--Last update from peer : (0) seconds, (770) msec

vPC Keep-alive parameters
--Destination : 172.16.34.253
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : management
--Keepalive udp port : 3200
--Keepalive tos : 192