Re: Need assistance configuring Peer-link between Nexus 7010s

wil_amaya · ‎10-21-2010

Hello all. I would like to know if someone can suggest or assist with the following:

I have a client that is building a new data center. My task is to configure vPC between two Nexus 7010 switches that will sit at opposite ends of the building. The building is massive and the distance between the two 7010s will not allow me to use Ethernet/Copper for the peer-link or L3 adjacencies. It must be fiber. I feel that this is a waste of 10gb ports as not much traffic crosses those 2 links and in addition the client wants full redundancy on everything so i'd be configuring in port-channels. That means that not only do i used 2 x 10GB interfaces for my peer-link but also 2 x 10Gb interfaces for the keep alive and 2 x 10GB interfaces for the L3 adjacency. Thats 4 additional 10GB ports I'd be using up for simple keep-alive and L3 traffic.

I have staged the equipment and have vPC up and running but with the peer-link on 10gb interfaces and the keep-alive and L3 adjacency on ethernet ports but i know that has to change. My question is this:

Can I run my L3 adjacency or keep-alive link through downstream switches that will connect to the 7010s via vPC. To simplify, the design looks as follows:

Po1 - peer-link between 7010s

Po2 - keep-alive

Po3 - L3 adjacency

Po4 - vPC 4, single Nexus 5548 closet switch.

So, using Po4, can I pass my keep alive or L3 traffic over that? Or at least one of them?

Thanks in advance and sorry if this sounds confusing.

Wil

Adam Casella · ‎10-21-2010

Hey Wil,

This should not be a waste of 10gb ports as it is a requirement for the VPC peer link. The reason for this is that if one the VPC member ports fail we want to have adequte bandwidth to continue forwarding traffic if it needs to pass the over the peer link (which is quite a likely scenario).

The keep alive link needs to be a layer 3 connection in its own VRF and will not pass any user data and does not need to high bandwidth. A 10 gb is a waste if there are other methods to connect to the two switches.

With that said, I would NOT recommend to run the keep alive link through any other device. This is designed to be a point to point link and if any drops occur on a device in the middle you could have issues with your VPC.

Unfortunately, any other method may result in unexepected consequences.

Thanks,

Adam

wil_amaya · ‎10-21-2010

Adam - hello and thank you for the reply. I agree that using 10GB interfaces for the peer-link is not a waste. In

fact its a requirement. What i thought was wasteful was using 10GB interfaces for the keep-alive and the L3 adjacency. So what i'm looking for is a way to configure my l3 and keep-alive through another switch. Any help or sample configs is apprecaited. Tahnks.

Wil

Adam Casella · ‎10-21-2010

Hey Wil,

Thinking about this a little deeper you could use the mgmt interfaces on the N7k for your VPC keep alive link. It is recommended to use a switch in between the two mgmt interfaces when used as the keep alive link.

The reason is that we want to make sure the mgmt link stays up if their is a failure on the n7k for the keepalive link when using the mgmt interfaces.

This would allow you to run further than 100M limited for copper based Ethernet. This would look like the following:

n7k(mgmt)-------switch-------(mgmt)n7k

This should only be done when using the mgmt interface. This will allow you to save your 10gb links.

Also, redundancy is not needed for the keep alive link, as this is a backup for the peer link to keep dual-actvie situations from occurring. If the keep alive link fails, but the peer link does not, the devices learn of eachother exsitence via the peer-link. This allows the two devices to continue to work until the fault in the keep alive is resolved.

Thanks,

Adam

wil_amaya · ‎10-21-2010

Adam, thanks again for the reply. Using the management interfaces for the keep-alive is a good idea. i will see if my client will allow that. I know they are hung up on redundancy for everything so we'll see.

Any thoughts on the L3 adjacency? One of our architects mentioned he thought he saw/heard/myth that you can run your L3 adjacency through your downstream vPC switches. But if the downstream are L2 switches how can that be possible? Thanks.

Wil

Atif Awan · ‎10-21-2010

wil_amaya wrote:

Adam, thanks again for the reply. Using the management interfaces for the keep-alive is a good idea. i will see if my client will allow that. I know they are hung up on redundancy for everything so we'll see.

Any thoughts on the L3 adjacency? One of our architects mentioned he thought he saw/heard/myth that you can run your L3 adjacency through your downstream vPC switches. But if the downstream are L2 switches how can that be possible? Thanks.

Wil

Wil,

Are you referring to L3 adjacency between the N7Ks? If yes then why not run it over the peer-link. The underlying assumption is that your peer-link is over dual 10-Gig links with each link on a separate linecard.

By the way point-to-point L3 over peer-link between two N7Ks is fine. What is not supported is running routing over a vPC interconnect with another entity (router/firewall).

Atif

wil_amaya · ‎10-21-2010

Atif - thanks for the response. Yes, I am referring to the L3 adjacency between the 7ks. And yes, we are running over Dual 10gb interfaces on seperate line cards. The peer-link is L2 so how would i run L3 adjacency over it. Can you provide a sample config please? Thanks.

wil

Atif Awan · ‎10-21-2010

wil_amaya wrote:

Atif - thanks for the response. Yes, I am referring to the L3 adjacency between the 7ks. And yes, we are running over Dual 10gb interfaces on seperate line cards. The peer-link is L2 so how would i run L3 adjacency over it. Can you provide a sample config please? Thanks.

wil

You do that by running routing over an SVI. For example, let's assume you want to use VLAN 300 for point-to-point OSPF routing between the two N7Ks. In this case you will carry VLAN 300 over the vPC peer-link and create an SVI on each N7K with a configuration like:

interface Vlan300
no shutdown
description *** SVI -for- OSPF-VLAN ***
no ip redirects
ip address IP_Address/30
ip ospf message-digest-key 1 md5
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0

no shutdown

Atif

Adam Casella · ‎10-22-2010

Hey Atif,

The issue with that configuration is that the VPC keepalive link needs to be in its own VRF and should be a point to point layer 3 interface. The reason for this is that it is suppose to be a back up link to the VPC link, to prevent dual active scenarios.

The Best bet for Wil at this case would be to use the mgmt interfaces (which are in the management vrf) and have it pass through a switch IF you use the mgmt interface for the keep alive link. The reason for this is that you must make sure that one side remains up if a faiure occurs on oppostie side.

You can the keep alive link through any other device as long as the link is on its own VRF on each end, however this is NOT supported, because of the increased likely hood of drops on the intermediate devices.

The best option (and really the only supported/valid option that I see) would be to use the mgmt interfaces as I mentioned above. This setup for the keepalive link is currently being run by numerous organizations in the field.

Thanks,

Adam

Atif Awan · ‎10-22-2010

acasella wrote:

Hey Atif,

The issue with that configuration is that the VPC keepalive link needs to be in its own VRF and should be a point to point layer 3 interface. The reason for this is that it is suppose to be a back up link to the VPC link, to prevent dual active scenarios.

The Best bet for Wil at this case would be to use the mgmt interfaces (which are in the management vrf) and have it pass through a switch IF you use the mgmt interface for the keep alive link. The reason for this is that you must make sure that one side remains up if a faiure occurs on oppostie side.

You can the keep alive link through any other device as long as the link is on its own VRF on each end, however this is NOT supported, because of the increased likely hood of drops on the intermediate devices.

The best option (and really the only supported/valid option that I see) would be to use the mgmt interfaces as I mentioned above. This setup for the keepalive link is currently being run by numerous organizations in the field.

Thanks,

Adam

Adam,

I am with you on the vPC Peer-Keepalive; my comment was actually for the Layer-3 adjacency between the N7Ks and not the peer-keepalive. Wil had mentioned that he had a requirement for running dynamic routing between the N7Ks and that is what I was referring to. There is no doubt that peer-keepalive should not share the state with peer-link as it defeats the purpose of the keepalive. Hope this clarifies the confusion.

Atif

wil_amaya · ‎10-22-2010

Guys, thank you both for your suggestions. i will go with Adams method for the keep-alive setup and with Atif's for the L3 adjacency. How do i give you both credit for the correct answer? Thanks.

Wil

wil_amaya · ‎12-13-2010

Atif, hello. I went with using the L3 over the peer-link and it worked fine, however, I still had my doubts about it so I opened a case with Cisco. they said you should not run an L3 connection over the peer-link. They also said it is not a supported configuration. Just an FYI for future reference in case anyone else runs across something similar. For now I've gone ahead and dedicated 10GB interfaces for the L3 adjacency and the keep-alive. When I get some other switches in place I will see if I can use the management interfaces for my keep-alive. Otherwise I'll have to just burn those 10GB ports. Thanks again for your help and suggestions.

WIl

Atif Awan · ‎12-13-2010

wil_amaya wrote:

Atif, hello. I went with using the L3 over the peer-link and it worked fine, however, I still had my doubts about it so I opened a case with Cisco. they said you should not run an L3 connection over the peer-link. They also said it is not a supported configuration. Just an FYI for future reference in case anyone else runs across something similar. For now I've gone ahead and dedicated 10GB interfaces for the L3 adjacency and the keep-alive. When I get some other switches in place I will see if I can use the management interfaces for my keep-alive. Otherwise I'll have to just burn those 10GB ports. Thanks again for your help and suggestions.

WIl

Will,

Layer-3 between Nexus 7Ks only over the peer-link is a supported configuration and I am not sure why you were advised otherwise. What is not supported is to run this in a shared topology where the Nexus 7Ks are also talking to other devices, like a pair of firewalls. My understanding was that you were going to use it only for the Nexus 7Ks point-to-point connectivitry and there should be no support issues with it. Can you please unicast me the case number so I can dig deeper into this?

Atif

wuhao_xiaotong · ‎07-08-2011

Hi Atif,have you found the answer about the Case what the TAC said? I met the same problem that my customer has the idea as you gave above , would it indeed a good option?