Ping from VxLAN distribute anycaste gateway(VPC) to one of Server.

[fly]

How can I ping from one of vxlan distribute anycast gateway router (VPC） to  directly connected subnet Server。

we config a vxlan network(SPINE and leaf bgp evpn), and using distribute anycast gateway on two switch formed by VPC.

due to VPC nature , two leaf switches formed by VPC, if I login to one of leaf switch and ping directly on this switch to directly connected Server, the PING maybe unsuccess, but login to another VPC switch , the PING task is success.

the problem caused by VPC nature, because downside switch made load balance by Port channel loadblance algorithm

How can I ping from vpc leaf switch  at this situation for management  task?

[Varun Jose]

you can follow below steps

1) spin up a loopback in tenant VRF,

2) give it a unique IP address,

3) advertise the same into l2vpn evpn(as a type-5 route)

4) use the advertise-pip and advertise-pip and advertise virtual-rmac under BGP and nve interface respectively which would make sure traffic to this loopback is attracted to the right VTEP; you can get more details here-> https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_01011.pdf

[fly]

Hi,Varun,

    I think this is a right solution. but I test it , not success,I will test gain.

   I sniffer some packet found some behaviors:

   I config as your said.    and ping from leaf1 to downside server, source ip is loopback's in correspond vrf.

    from capture one leaf1 and leaf2 , I found return packet come from server pass leaf2 (not leaf1), and i config advertise pip and rmac.  and bgp routing entry is correct.

   and I found I can capture return packet on leaf1 underlay interface( connect to spine), 

    but ping is not succcess ,

   I will test again

   thank you!

Jere

[fly]

I test it again , 

I have ileaf 01 and ileaf02  formed by vpc. config as a VPC VTEP, bgp evpn reflector client

and config vrf zsc, and hp blad sever connect by VPC 17. and distribute gateway on two vpc VEP ileaf01 ileaf02

two vteps using interface loopback1 secondary ip 10.9.1.4 as vpc VTEP anycast IP.

 and I config interface loopback 1 primary ip 10.9.1.21  on ileaf01  and interface loopback1 primary ip 10.9.1.22 on ileaf02

and turn on advertise pip rmac.

I create loopback 22 ip 22.1.1.1/32 put into vrf zsc on ileaf01 and loopback 22 ip 22.1.1.2/32 on ileaf02,put into same vrf.

the purpose of my test  is to create a ping test on ileaf01 to downside server(connected by VPC),  or ping test on ileaf02 to downside server(connected by same VPC) to do some management job like we run in tradition network.

I made a capture on ileaf02 control plane and ping from 22.1.1.1 to 22.1.1.2 on ileaf01 ,can't success.

I found packet can arrive at ileaf02 control plane, but outer vxlan packet's  source ip is 10.9.1.4(ileaf01 ileaf02 vpc anycast ip) not ileaf01 primary ip (10.9.1.21) destination is 10.9.1.22(ileaf 02 loopback1 primary IP),learn by bgp due to turn on advertise pip.

but I didn't find any reply icmp packet from ileaf02 control plane sending back .

I think because packet outer vxlan source ip is 10.9.1.4 is VTEP anycast ip ， and ileaf02 also has same VPC vtep anycast IP 10.9.1.4 under interface loopback 1, So due to this reason , control plane think this packet generated on his own and vxlan outer destination ip is on his own(10.9.1.22), so control plane drop this packet,I can't prove this.

So your solution can detect remote vrf endpoint under remote VTEP，not ping on  one local VPC VTEP to directly connected VPC server 。

may be I can ping  server connected by one VPC on remote VPC VTEP,

I create  loopback interface on remote VTEP bleaf01 and bleaf02, config different IP, and then  ping from remote VPC VTEP bleaf01 and bleaf02, this ping test cross vxlan spine  and hit destination server, destination server can made portchannel load balance , and choose different physical link , and in this way I can test two physical link of same VPC

this confuse me， can't find best solution.

 if destination is type 5 route, and turn on advertise pip , cisco can change vxlan source ip to interface loopback 1 primary ip , not vpc anycast ip , I think is a solution.

now I have nothing I can Do, may be I can create a tradition vrf vlan interface passthrough peer-link ,  But I can't creat one interface per VRF, because there are more than 200 VRFS.

another solution is using LACP actice mode to detect physical link failure.

thank you

Jere

[fly]

I made a test , but can't work

I ping from server 10.20.72.24 to 22.1.1.2(loopback22 on leaf02),actually this ping hit leaf01 vpc physical interface first,

and encapsulate into vxlan then send to leaf02 through spine, I can capture this traffic arrived leaf02 underlay interface.

But I can' see any return icmp reply traffic.

but i can ping from remote border leaf switch.[

below is my test

I create loopback22 on both leaf01 and leaf02 and put into vrf ZSC
on leaf01 interface loopback22 ip is 22.1.1.1/32

on leaf01 interface loopback22 ip is 22.1.1.2/32

I can bgp can update this loopback22 host route come from anothoer vpc vtep primary ip address.

but I find a problem first, for example:
on leaf01 I ping leaf02's loopback22 ip 22.1.1.2 and source interface ip is leaf01 loopback22 ip 22.1.1.1.
such ping put into vrf ZSC
but I can't ping it successfully

but I can ping loopback22 on leaf01 and leaf02 on remote two border leaf switch successfully

I made another test.
I ping 22.1.1.2( leaf02 loopback22 ip) on server (vpc connected to two leaf switches).
I can see ping traffic can come into leaf01 port-channel interface and
I can see this ping can come into underlay interface on leaf02, but I didn't see any icmp reply send from leaf02 any interface.include underlay interface ,and vpc downside interface(connect to server).

that weird , I didn't capture packet on peer-link interface.

here is some capture on leaf02 underlay interface
please see my notes under below capture:

Internet Protocol, Src: 10.9.1.4 (10.9.1.4), Dst: 10.9.1.22 (10.9.1.22)
///////////here I found something: the source ip address is leaf01,leaf02 vpc anycast IP ,not primary ip under same loopbak1 interface.

Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 110
Identification: 0x8000 (32768)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 254
Protocol: UDP (0x11)
Header checksum: 0x2653 [correct]
[Good: True]
[Bad : False]
Source: 10.9.1.4 (10.9.1.4)
// 10.9.1.4 is leaf01 leaf02 vpc vtep anycast ip configured under interface loopback1
Destination: 10.9.1.22 (10.9.1.22)
//10.9.1.22 is leaf02 primary ip configured under interface loopback1 .

User Datagram Protocol, Src Port: 56233 (56233), Dst Port: 4789 (4789)
Source port: 56233 (56233)
Destination port: 4789 (4789)
Length: 90
Checksum: 0x0000 (none)
Good Checksum: False
Bad Checksum: False
Virtual eXtensible Local Area Network
Flags: 0x08
0... .... = Reserved(R): False
.0.. .... = Reserved(R): False
..0. .... = Reserved(R): False
...0 .... = Reserved(R): False
.... 1... = VXLAN Network ID(VNI): True
...0 .... = Reserved(R): False
...0 .... = Reserved(R): False
...0 .... = Reserved(R): False
Reserved: 0x000000
VXLAN Network Identifier (VNI): 130022
Reserved: 0
Ethernet II, Src: 02:00:0a:09:01:04 (02:00:0a:09:01:04), Dst: d4:c9:3c:b5:3c:17 (d4:c9:3c:b5:3c:17)
Destination: d4:c9:3c:b5:3c:17 (d4:c9:3c:b5:3c:17)
Address: d4:c9:3c:b5:3c:17 (d4:c9:3c:b5:3c:17)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: 02:00:0a:09:01:04 (02:00:0a:09:01:04)
Address: 02:00:0a:09:01:04 (02:00:0a:09:01:04)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
Type: IP (0x0800)
Internet Protocol, Src: 10.20.72.24 (10.20.72.24), Dst: 22.1.1.2 (22.1.1.2)

// this is ping from server to loopback22 on leaf02 in vrf ZSC.

Version: 4
Header length: 20 bytes
--More--200 packets captured
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0x05e1 (1505)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 127
Protocol: ICMP (0x01)
Header checksum: 0xccb1 [correct]
[Good: True]
[Bad : False]
Source: 10.20.72.24 (10.20.72.24)
Destination: 22.1.1.2 (22.1.1.2)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0 ()
Checksum: 0x4cc9 [correct]
Identifier: 0x0001
Sequence number: 146 (0x0092)
Data (32 bytes)

server 10.20.72.24 ping to 22.1.1.2(loopback22 on leaf02) come into leaf01 physical interface E1/17(vpc 17)
and encapsulate in a vxlan packet ,outer source ip is leaf01 vpc vtep anycast ip 10.9.1.4(not primary
ip), destination is leaf02 primary ip 10.9.1.22,this traffic can hit leaf02 through underlay interface ,
I think leaf02 decapusulate this traffic from vxlan packet , and hit loopback22 ip 22.1.1.2 in same vrf ZSC.
and return a icmp reply:
source ip is 22.1.1.2 desination ip is 10.20.72.24(in vrf ZSC).
10.20.72.24 under directly connected subnet on leaf02 ,and should send to vpc physical interface directly .

but i didn't capture this return icmp reply on vpc physical interface on either leaf01 leaf02 switch.

[fly]

Hi,Varun,

    I test it again during last weekend. but  I found solution you provided is not working in my situation

    I start a ping from ileaf01 by using source ip 22.1.1.1( in a test VRF loopback 22) to downside directly connected server 10.20.74.24(VPC,same VRF), 

    I can find sever return packet was coming into ileaf02(vpc peer of ileaf01) , and then ileaf02 made a Vxlan encapsulation by using outer vxlan vtep sources address 10.9.1.4 ( vpc vtep anycast address, not ileaf02 primary ip address) and vxlan outer  destination vtep ip is 10.9.1.21(ileaf01 primary ip under same loopback 1 interface of vpc vtep anycast ip), 

   I also can capture packet on ileaf01 control plane , ileaf01 control plance actually received this server return icmp reply packet from vxlan underlay interface from ileaf01, I can see this vxlan packet coming into ileaf01 control plane. But ping  failed.

    I think the pakcet send from ileaf02( encapsulated in vxlan) by using ileaf01 ileaf02 vpc anycast IP ( same share vpc vtep ip for ileaf01,ileaf02) as source ip address , and destination ip is ileaf01 vtep primary ip ,   ileaf01 maybe think this vxlan packet actually send from ileaf02 sending from his own and will hit his own primary ip , so ileaf01 drop this packet.

   thank you

Jere

[Varun Jose]

Could you reply on below queries;

- what is the NXOS version on n9ks

- what exact model are those n9ks

btw, i tested on 92160 running 9.2(3) and it worked perfectly fine. 

[fly]

our spine is 9508 , I config as pure spine

leaf is 93180YC-EX ,version is 

Software
BIOS: version 07.65
NXOS: version 7.0(3)I7(6)
BIOS compile time: 09/04/2018
NXOS image file is: bootflash:///nxos.7.0.3.I7.6.bin

thank you

Jere

[melchib]

Was there ever a resolution to this?

We have the exact same issue.

[fly]

no, I open a case

tac suggest me to create a special vrf and confile unique ip address on l3 interface in this VRF, and config speical vlan interface on blade server chassis， then test this l3 interface from nexus leaf special vrf l3 interface