VPC link and keep-alive setup

[Ozy]

Hello. 

I have 2 Nexus n9k and this is gonna be my first VPC setup. 

I have 4 x 25Gbit links between the two switch because this switches gonna serve for storage devices, I need speed and low latency. I did not use 100G uplinks because I need them for different switch connections. Thats why I used 4x25Gbit but I want to be sure is it logical or not. 

As I read, I need a link (or two link for redundancy) for VPC keep-alive but I did not understand why we need seperate link for keep-alive.

I'm not experienced network engineer but as a developer and system engineer I believe we should be able to use these links as peer-link and also keep-alive link. We just need two different layer. 

1- Layer 2: vpc peer-link on 4x interface 

2- Layer 3: vpc keep-alive link on 4x interface with port-channel

My question: Is there any way to use the 4 x 25Gibt VPC links also as keep-alive?

With this setup, we will be able to use all of the interface for peer and keep-alive so it is more logical and safe. I don't know but the keep-alive network package must be only a tcp package nothing more and the communication speed is 1 sec or similar I suppose. So I really do not understand why we have to use seperate keep-alive link. what is the logic behind this???

[M02@rt37]

Hello @Ozy,

In a Nexus N9K VPC setup, using 4 x 25Gbit links between the switches is a valid configuration. The choice of using 4 x 25Gbit links instead of 100G uplinks depends on your specific requirements and available resources, such as the need to allocate 100G uplinks for other switch connections.

Regarding the VPC keep-alive link, it is recommended to have a separate link or links dedicated solely for VPC keep-alive traffic. The purpose of the keep-alive link is to provide a reliable and independent communication path between the two Nexus switches for the VPC peer-link heartbeat and VPC consistency checks.

The keep-alive link is crucial for detecting failures and ensuring proper coordination between the switches in the VPC. It helps prevent scenarios where the VPC peers might incorrectly assume that the other peer is down due to other network issues affecting the data traffic. By having a separate link for keep-alive, it helps maintain the integrity and stability of the VPC operation.

It is generally recommended to dedicate separate links specifically for this purpose. This separation ensures that the keep-alive traffic is isolated and does not interfere with or get affected by the regular data traffic flowing over the VPC links. Using dedicated links for keep-alive also provides better control and visibility over the VPC keep-alive traffic, making troubleshooting and monitoring easier. Therefore, it is recommended to follow the best practice of using separate links for VPC keep-alive traffic, even if it means utilizing additional ports or interfaces on the switches.

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/218333-understand-and-configure-nexus-9000-vpc.html

[Ozy]

Thank you so much for this beatiful, simple, educating answer. 

With this setup, we will be able to use all of the interface for peer and keep-alive so it is more logical and safe to me. I don't know but the keep-alive network package must be only a tcp package nothing more and the communication speed is 1 sec or similar I suppose. So I really do not understand why we have to use seperate keep-alive link. what is the logic behind this???

Then I have to ask different questions:

I have 100/Mbit management switch and the management ports connected there. 

1- Is it logical to use this for keep-alive because the keep-alive does not need too much bandwidth ?

2- Using third switch does effect the speed or create unexpected latency or weird things like this? 

3- Can we gain speed by using direct link between two vpc peer ? I only have these 4 x 25gbit links and for only keep-alive its not logical to use 25Gbit link. What should I do? 

4- If I lose the keep-alive link but the switches and peer-links are alive whats gonna happen? 

[M02@rt37]

Hello @Ozy,

You're very welcome.

1.Using the Management Switch for Keep-Alive: While it is technically possible to use the management switch for the VPC keep-alive communication, it is generally not recommended. The management switch is typically reserved for out-of-band management traffic and may have its own limitations and potential points of failure. It is preferable to have a dedicated, separate link for keep-alive traffic to ensure its isolation and reliability.

2.Using a Third Switch: VPC domain = 2 nexus only.

3.Using Direct Links between VPC Peers: In a traditional VPC setup, utilizing direct links between the VPC peers is not possible. The VPC peer-link connects the two Nexus switches and forms the core of the VPC. It provides the necessary control plane communication and synchronization between the peers. Utilizing the 25Gbit links for both peer-link and keep-alive is the appropriate design choice.

4.Impact of Losing the Keep-Alive Link: In the event of a failure or loss of the keep-alive link, while the VPC peer-link and switches remain operational, the VPC domain may encounter issues. The absence of a functioning keep-alive link can prevent the switches from exchanging critical control plane messages, potentially leading to instability, synchronization problems, or even a complete VPC failure. It is essential to ensure the reliability and availability of the keep-alive link to maintain a healthy VPC setup.

--Using the same interfaces for both peer-link and keep-alive, it is not recommended. The separate keep-alive link provides isolation, stability, and resilience for the control plane communication of the VPC setup. It is best to follow the standard guidelines and ensure the availability and reliability of the keep-alive link to maintain a robust VPC environment.

Furthermore, the peer keepalive does not need to be a physical link. It just needs to be connectivity at L3. For example, I have a routed access layer using Nexus and they establish OSPF relationships with the upstream distribution layer.  I use the loopback IPs as my peer keepalive endpoints.  It works perfectly!

[MHM Cisco World]

No you need separate and the reason is 

Keepalive is l3 link

Peer-link is l2 link

[MHM Cisco World]

If peer-link is down there is specific behavior for both NSK

If the keep-alive is down there is other behavior 

Merge both in one link I never see and I dont predict the behavior of NSK' but sure this lead to split brain.

Why you insist to merge both?

[Ozy]

I'm a hyperconverged subsystem engineer so I always intend to use all the resources and combine them together nicely to reduce cost and gain redundancy at the same time. The keep-alive package is very small and the latency and speed requirement is not that important. I "guess" it is only using for to understand other end current state in every few seconds. 

For example lets create a port channel with 4 interface and use all of them for two things;
1- peer links -> layer2
2- keep alive link -> layer3 

it is possible but I read the datasheet and they say "do not do that" so I stop thinking on this.

Today I configured my switches as;
1- 3 x 25Gbit interface for peer-links --> port-channel 
2- 1 x 25Gbit interface for keep-alive -->  and internal interface ip for both end as default vrf (I'm not sure about this, I think my default vrf should be my main vlan. I have to learn this)

Let me share my setup:

# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 100
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 39
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Enabled
Virtual-peerlink mode : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po100 up 1

# sh vpc peer-keepalive

vPC keep-alive status : peer is alive
--Peer is alive for : (47037) seconds, (437) msec
--Send status : Success
--Last send at : 2023.07.07 03:30:03 521 ms
--Sent on interface : Eth1/45
--Receive status : Success
--Last receive at : 2023.07.07 03:30:03 522 ms
--Received on interface : Eth1/45
--Last update from peer : (0) seconds, (109) msec

vPC Keep-alive parameters
--Destination : X.X.X.2
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : default
--Keepalive udp port : 3200
--Keepalive tos : 192

# show port-channel summary interface po100
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
100 Po100(SU) Eth LACP Eth1/46(P) Eth1/47(P) Eth1/48(P)

# show interface e1/45-48 status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/45 -- connected routed full 25G SFP-H25GB-CU3M
Eth1/46 VPC Peer-Link connected trunk full 25G SFP-H25GB-CU3M
Eth1/47 VPC Peer-Link connected trunk full 25G SFP-H25GB-CU3M
Eth1/48 VPC Peer-Link connected trunk full 25G SFP-H25GB-CU3M

Everything works as expected and my switch VPC configuration is almost complete.
Now I want to create an internal default gateway for my main vlan. I don't know yet, I'm planning to learn tomorrow. 

The questions I have now: 
1- I think the internal default gateway should be on the VPC or VRF for acting as one.

2- Do I need any special load-balance features because the best and fastest way is balancing via gateway. But I'm gonna use these for storage devices so if the decision algorithm will cause extra latency, it will be bad for me. I have to learn and find some parameters to get minimum latency possible.

[MHM Cisco World · Accepted solution]

vPC Failure Scenarios - Impact and Solution - Cisco vPC Guide (letsconfig.com) 
this two behave of vPC I mention before.
NOW since you separate keep-alive than Peer-link 
that great 
NOW 

The questions I have now: 
1- I think the internal default gateway should be on the VPC or VRF for acting as one.
 2- Do I need any special load-balance features because the best and fastest way is balancing via gateway. But I'm gonna use these for storage devices so if the decision algorithm will cause extra latency, it will be bad for me. I have to learn and find some parameters to get minimum latency possible.

Answer Config VLAN SVI in both NSK 
config HSRP 
make VIP of HSRP group the GW of host 
NOTE:- dont forget to add peer-gateway under the vpc domain

[Ozy]

The visual and information is awesome and easy to understand. Thank you so much. 

I have only 1 vlan which is "vlan 1" and the subnet is "/20" 
In this case, I believe I don't need VLAN SVI and also HSRP right?
My servers are "802.3ad & layer2+3" LACP will use layer2 and if I'm not wrong, actually I don't need routing internally,

---

@MHM Cisco World wrote:

NOTE:- dont forget to add peer-gateway under the vpc domain

---

What is the usage cases and what is the proper way to add? 

[MHM Cisco World]

this design I think is what you need