Re: ping problem in vpc anycast vtep environment

ez9 · ‎05-08-2020

Hello,

I have a topology(attached) using 2x Cisco nexus 9396px switches in vpc mode. I enabled vxlan feature and I implement anycast vtep.

The anycast IP is 192.168.93.93. The switchA has the 192.168.93.101 and the SwitchB the IP 192.168.93.102.

Also, I have a different loopbacks for bgp evpn peering.

Between the 2 switch I have a L3 dedicated link (e1/5) which running OSPF for underlay loopback reachability.

I have advertise-pip and advertise virtual-rmac enabled.

The problem is that client1 cannot ping looopback on R1 router. The switch1 and switch2 have a route in vrf table for this loopback IP. I tried to capture the traffic and I can see that the packet reaches to switch2 but it not forwarding to the R1 via interface e1/3.

I also tried to create a seperate loopback in each switch in vrf (switchA: 10.10.10.93, switchB: 10.10.10.193). I announced these IPs in BGP and i tried to ping between these IPs. Unfortunatelly the ping is failing again. In the packet capture I can see the packets reaching the other switch with VXLAN encapsulation, but the reply is never generated.

Now the switches running the version 7.0(3)I7(7). I tried to upgrade to 7.0(3)I7(8) but with the same results.

Can you provide any insights on this?

f00z · ‎05-09-2020

Can you share the configuration? Your explanation is a little hard to follow.

Thanks!

ez9 · ‎05-09-2020

Hello,

I attach the configurations of the two switches. The router has a loopback address which announce to SW2 via eBGP.

f00z · ‎05-09-2020

What is the client IP and which loopback is it trying to ping?

I assume the default gateway on the client is pointed to the anycast gw? this should work no problem as long as the loopback int is in the same vrf as the anycast gw int (on the same switch, i.e. if the client is connected to switch 2 and pings the loopback on switch 2, this works 100% as it's not even related to vxlan)

Pinging the loopback on switch1 from a client on switch 2 is where you need to redistribute it into bgp

It looks like you already did this in your config. so you need to verify on both switches:

show bgp ipv4 unicast vrf Customers

show bgp l2vpn evpn vrf Customers

show bgp l2vpn evpn route-type 5

See if the route is there (the loopback)

Only odd things I see is you have mac addresses set on the uplink interface.

You don't have peer-gateway, peer-switch enabled on VPC which is best practice.

Ahh i read it for the 5th time and thought you were referring to pinging the loopbacks on sw1/2 , but you want to ping the one on r1 which you didn't include the config for. I'd like to see the routing tables.

Can you ping r1 loopback from a client on switch 2? that wouldn't even use vxlan or evpn as it's in the same vrf locally connected. Once you verify that works we can move on to switch 1.

ez9 · ‎05-09-2020

The client1 has the IP: 192.168.70.10 and client2 the IP: 192.168.70.20. They are using the anycast gateway 192.168.70.1.

The loopback behind R1 is the 10.88.88.88. I don't get any response when ping from the client1 to this loopback.

From client1 I can ping the loopback in SW1 (10.10.10.93) but not the loopback in SW2 (10.10.10.193). I can capture the VXLAN packet coming to SW2 but the SW2 never sends a response back.

On SW1:

show bgp ipv4 unicast vrf Customers

   Network            Next Hop            Metric     LocPrf     Weight Path
*>l10.10.10.93/32     0.0.0.0                           100      32768 i
*>i10.10.10.193/32    192.168.93.102                    100          0 i
*>i10.88.88.0/24      192.168.93.102           0        100          0 65502 i
* i192.168.70.0/26    192.168.93.102                    100          0 i
*>l                   0.0.0.0                           100      32768 i

show bgp l2vpn evpn vrf Customers

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.0.93.101:33047    (L2VNI 28000)
*>l[2]:[0]:[0]:[48]:[0050.7966.6803]:[0]:[0.0.0.0]/216
                      192.168.93.93                     100      32768 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6803]:[32]:[192.168.70.10]/272
                      192.168.93.93                     100      32768 i
*>l[3]:[0]:[32]:[192.168.93.93]/88
                      192.168.93.93                     100      32768 i

Route Distinguisher: 10.0.93.102:3
*>i[5]:[0]:[0]:[24]:[10.88.88.0]:[0.0.0.0]/224
                      192.168.93.102           0        100          0 65502 i
*>i[5]:[0]:[0]:[26]:[192.168.70.0]:[0.0.0.0]/224
                      192.168.93.102                    100          0 i
*>i[5]:[0]:[0]:[32]:[10.10.10.193]:[0.0.0.0]/224
                      192.168.93.102                    100          0 i

Route Distinguisher: 10.0.93.101:3    (L3VNI 900001)
*>l[2]:[0]:[0]:[48]:[5000.0001.0007]:[0]:[0.0.0.0]/216
                      192.168.93.93                     100      32768 i
*>i[5]:[0]:[0]:[24]:[10.88.88.0]:[0.0.0.0]/224
                      192.168.93.102           0        100          0 65502 i
* i[5]:[0]:[0]:[26]:[192.168.70.0]:[0.0.0.0]/224
                      192.168.93.102                    100          0 i
*>l                   192.168.93.101                    100      32768 i
*>l[5]:[0]:[0]:[32]:[10.10.10.93]:[0.0.0.0]/224
                      192.168.93.101                    100      32768 i
*>i[5]:[0]:[0]:[32]:[10.10.10.193]:[0.0.0.0]/224
                      192.168.93.102                    100          0 i

show bgp l2vpn evpn route-type 5

BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.93.102:3
BGP routing table entry for [5]:[0]:[0]:[24]:[10.88.88.0]:[0.0.0.0]/224, version 2310
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path
             Imported to 2 destination(s)
  AS-Path: 65502 , path sourced external to AS
    192.168.93.102 (metric 2) from 10.0.93.102 (10.0.93.102)
      Origin IGP, MED 0, localpref 100, weight 0
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0002.0007

  Path-id 1 not advertised to any peer
BGP routing table entry for [5]:[0]:[0]:[26]:[192.168.70.0]:[0.0.0.0]/224, version 2295
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path
             Imported to 2 destination(s)
  AS-Path: NONE, path sourced internal to AS
    192.168.93.102 (metric 2) from 10.0.93.102 (10.0.93.102)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0002.0007

  Path-id 1 not advertised to any peer
BGP routing table entry for [5]:[0]:[0]:[32]:[10.10.10.193]:[0.0.0.0]/224, version 61
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path
             Imported to 2 destination(s)
  AS-Path: NONE, path sourced internal to AS
    192.168.93.102 (metric 2) from 10.0.93.102 (10.0.93.102)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0002.0007

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.0.93.101:3    (L3VNI 900001)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.88.88.0]:[0.0.0.0]/224, version 2311
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path
             Imported from 10.0.93.102:3:[5]:[0]:[0]:[24]:[10.88.88.0]:[0.0.0.0]/224 
  AS-Path: 65502 , path sourced external to AS
    192.168.93.102 (metric 2) from 10.0.93.102 (10.0.93.102)
      Origin IGP, MED 0, localpref 100, weight 0
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0002.0007

  Path-id 1 not advertised to any peer
BGP routing table entry for [5]:[0]:[0]:[26]:[192.168.70.0]:[0.0.0.0]/224, version 2296
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn

  Path type: internal, path is valid, not best reason: Weight
             Imported from 10.0.93.102:3:[5]:[0]:[0]:[26]:[192.168.70.0]:[0.0.0.0]/224 
  AS-Path: NONE, path sourced internal to AS
    192.168.93.102 (metric 2) from 10.0.93.102 (10.0.93.102)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0002.0007

  Advertised path-id 1
  Path type: local, path is valid, is best path
  AS-Path: NONE, path locally originated
    192.168.93.101 (metric 0) from 0.0.0.0 (10.0.93.101)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0001.0007

  Path-id 1 advertised to peers:
    10.0.93.102    
BGP routing table entry for [5]:[0]:[0]:[32]:[10.10.10.93]:[0.0.0.0]/224, version 15
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path
  AS-Path: NONE, path locally originated
    192.168.93.101 (metric 0) from 0.0.0.0 (10.0.93.101)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0001.0007

  Path-id 1 advertised to peers:
    10.0.93.102    
BGP routing table entry for [5]:[0]:[0]:[32]:[10.10.10.193]:[0.0.0.0]/224, version 65
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path
             Imported from 10.0.93.102:3:[5]:[0]:[0]:[32]:[10.10.10.193]:[0.0.0.0]/224 
  AS-Path: NONE, path sourced internal to AS
    192.168.93.102 (metric 2) from 10.0.93.102 (10.0.93.102)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 900001
      Extcommunity: RT:65500:900001 ENCAP:8 Router MAC:5000.0002.0007

  Path-id 1 not advertised to any peer

f00z · ‎05-09-2020

Hmm it's not being put where it should be.

show l2route mac-ip all

show forwarding ipv4 route vrf Customers

f00z · ‎05-09-2020

I really think you need the peer-gateway / peer-switch in the vpc domain config. I've never tried it without those, that could be it, there's some weirdness going on. At least put those in and continue. The loopbacks on sw1/sw2 should be totally pingable from any client on either sw1 or sw2 , regardless of R1.. So first get that working.

ez9 · ‎05-10-2020

I added the commands (peer-gateway, peer-switch). The result is the same.

The client1 can ping client2. So, the VXLAN bridging is working.

The client1 can ping the loopback at SW1, but not the loopbacks in SW2 and R1.

The client2 can ping the loopback in SW2 and also loopback at R1, but not the loopback in SW1.

From SW1 I cannot ping the loopback in SW2.

show l2route mac-ip all

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link 
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated 
Topology    Mac Address    Prod   Flags         Seq No     Host IP         Next-
Hops      
----------- -------------- ------ ---------- --------------- ---------------
280         0050.7966.6803 HMM    --            0          192.168.70.10  Local 
         
280         0050.7966.680a HMM    --            0          192.168.70.20  Local

show forwarding ipv4 route vrf Customers

IPv4 routes for table Customers/base

------------------+-----------------------------------------+----------------------+-----------------+-----------------
Prefix            | Next-hop                                | Interface            | Labels          | Partial Install 
------------------+-----------------------------------------+----------------------+-----------------+-----------------
127.0.0.0/8          Drop                                      Null0        
10.10.10.93/32       Receive                                   sup-eth1
10.10.10.193/32      192.168.93.102     nve1    
10.88.88.0/24        192.168.93.102     nve1             
192.168.70.0/26       Attached                                  Vlan280
192.168.70.0/32       Drop                                      Null0
192.168.70.1/32       Receive                                   sup-eth1
192.168.70.10/32      192.168.70.10                              Vlan280             
192.168.70.63/32      Attached                                  Vlan280

ez9 · ‎05-10-2020

I enabled the two commands (peer-gateway, peer-switch) but the result is the same.

From client1 I can ping SW1 loopback but not loopbacks on SW2 and R1.

From client2 I can ping loopbacks on SW2 and R1 but not this in SW1.

I can ping from client1 to client2, so the VXLAN bridging is working.

I cannot ping from SW1 loopback to SW2 loopback. In SW2 I capture the request but the reply is not generated.

What do you mean with "it's not being put where it should be."?

show l2route mac-ip all

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link 
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated 
Topology    Mac Address    Prod   Flags         Seq No     Host IP         Next-
Hops      
----------- -------------- ------ ---------- --------------- ---------------
280         0050.7966.6803 HMM    --            0          192.168.70.10  Local 
         
280         0050.7966.680a HMM    --            0          192.168.70.20  Local

show forwarding ipv4 route vrf Customers

------------------+-----------------------------------------+----------------------+-----------------+-----------------
Prefix            | Next-hop                                | Interface            | Labels          | Partial Install 
------------------+-----------------------------------------+----------------------+-----------------+-----------------
127.0.0.0/8          Drop                                      Null0         
10.10.10.93/32       Receive                                   sup-eth1
10.10.10.193/32      192.168.93.102     nve1    
10.88.88.0/32        192.168.93.102     nve1                            
192.168.70.0/26       Attached                                  Vlan280
192.168.70.0/32       Drop                                      Null0
192.168.70.1/32       Receive                                   sup-eth1
192.168.70.10/32      192.168.70.10                              Vlan280             
192.168.70.63/32      Attached                                  Vlan280

show ip route vrf Customers

IP Route Table for VRF "Customers"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.10.10.93/32, ubest/mbest: 2/0, attached
    *via 10.10.10.93, Lo888, [0/0], 09:04:29, local
    *via 10.10.10.93, Lo888, [0/0], 09:04:29, direct
10.10.10.193/32, ubest/mbest: 1/0
    *via 192.168.93.102%default, [200/0], 09:04:29, bgp-65501, internal, tag 65501 (evpn) segid: 900001 tunnelid: 0xc0a85d66 encap: VXLAN
 
10.88.88.0/24, ubest/mbest: 1/0
    *via 192.168.93.102%default, [200/0], 08:43:24, bgp-65501, internal, tag 65502 (evpn) segid: 900001 tunnelid: 0xc0a85d66 encap: VXLAN
 
192.168.70.0/26, ubest/mbest: 1/0, attached
    *via 192.168.70.1, Vlan280, [0/0], 09:04:29, direct
192.168.70.1/32, ubest/mbest: 1/0, attached
    *via 192.168.70.1, Vlan280, [0/0], 09:04:29, local
192.168.70.10/32, ubest/mbest: 1/0, attached
    *via 192.168.70.10, Vlan280, [190/0], 09:00:12, hmm
192.168.70.20/32, ubest/mbest: 1/0, attached
    *via 192.168.70.20, Vlan280, [190/0], 00:39:14, hmm

f00z · ‎05-10-2020

Need to run those commands on both switches, as you see the 'show forwarding' is what is actually programmed in the hardware, and the 10.88.88.0 is a /32 and not a /24. Although you should be able to ping both loopbacks from client1, the sw2 is probably trying to reply to client1 over the peer link (since it's an orphan port) and there might be something funky going on there. Look at the arp tables, forwarding tables, l2route mac and mac-ip on both switches.

So we need to see what is programmed in the hardware on both sw1 and sw2 to see why it's not replying.

Why don't you try it without VPC? See if that's the reason it isn't working, start from basic non vpc config and then switch to vpc config.

Did you include the ENTIRE config in your attachment? going out on a limb but did you do the 'hardware access-list tcam region arp-ether 256 double-wide ' ?

ez9 · ‎05-10-2020

Thanks for answer. The loopback is announced from R1 as /24 (the IP in loopback is 10.88.88.88 255.255.255.0).

I also suppose that the problem is somewhere in vpc and with the connectivity via peerlink.

(For this reason I tried to use physical link for L3 connectivity between the switches and not a vPC VLAN as the 9396 not supports the command 'system nve infra-vlans' so I don't know if peering via vPC vlan is support for my platform.)

I forgot to mention (and I am sorry about it :-) ) that this config is taken from virtual lab (using Nexus 9000v). So the command 'hardware access-list tcam region arp-ether 256 double-wide' is not available there. I have the same setup on physical switches but it hard to make changes in production enviroment. So, I replicated the config to a virtual lab in order to investigate this problem. Due to virtual lab the 'show forwarding' command isn't available. The output is taken from the physical switches.

Without the vPC the communication is working as expected but in this case the topology and the technologies that used are very different from the original scenario (vPC, anycast VTEP etc).

I tried to use NXOS 9.2.x and 9.3.x but the result remains the same. Anyway, I can't conclude if this is a 'bug', or if I missing something.

f00z · ‎05-10-2020

Could be cfs isn't distributing the arp entry to the other switch.. because what should happen when you ping sw2 from the client on sw1, the path should be like this:

client -> sw1 -> vxlan encap to sw2 rmac, sw2 decaps it (since it's going to sw2 rmac) and does a lookup in the vrf associated with the vni , finds that it is destined to the loopback int, and replies from that loopback by doing a lookup in the vrf associated with the loopback to find dst ip nexthop, finds the ip and associated l2route in its local database (because CFS synched up the arp entries from sw1/2 from the 'ip arp synchronize' command in VPC, finds arp entry in the vlan280 on sw2 and sends it out vlan 280 -> peer link -> sw1 -> client 1

So either, it's being blocked on the peer link? or, CFS isn't synching the arp entries. VPC fabric peering fixes all this mess but unfortunately the older switches don't support it.

So, verify the arp entries on both sides, verify cfs, sniff the traffic and see what's going over the peer link.

ez9 · ‎05-11-2020

I done some tests with physical switches but I am getting some 'weird' results.

I am trying to ping from Client1(192.168.70.10) to loopback in SW2 (10.10.10.193) but am getting 'TTL expired in transit.' I capture this traffic and the srcmac is the physical MAC from SW1. So, I suppose tha the SW1 cannot find the way to forward the packet using VXLAN to SW2.

I tried to create an SVI between in two switches in vrf Customers and to establish iBGP. With this way the communication is ok. But in this case the VXLAN is not using. Maybe there is a problem with VXLAN routing?

Some command outputs: (vlan99 is temporary for tests).

SW1:

sh forwarding vrf Customers

slot  1
=======


IPv4 routes for table Customers/base

------------------+-----------------------------------------+----------------------+-----------------+---------------
--
Prefix            | Next-hop                                | Interface            | Labels          | Partial Instal
l
------------------+-----------------------------------------+----------------------+-----------------+---------------
--
127.0.0.0/8          Drop                                      Null0
10.10.10.93/32       Receive                                   sup-eth1
10.10.10.193/32      192.168.93.102     nve1
10.88.88.0/24        192.168.93.102     nve1
10.93.1.2/31         Attached                                  Vlan99
10.93.1.2/32         Receive                                   sup-eth1
10.93.1.3/32         10.93.1.3                                 Vlan99
192.168.70.0/26      Attached                                  Vlan280
192.168.70.0/32      Drop                                      Null0
192.168.70.1/32      Receive                                   sup-eth1
192.168.70.10/32     192.168.70.10                             Vlan280
192.168.70.63/32     Attached                                  Vlan280

sh ip arp suppression-cache local

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface

Ip Address      Age      Mac Address    Vlan Physical-ifindex    Flags

192.168.70.10   00:03:26 b8ac.6f86.b153  280 Ethernet1/10        L

sh ip route vrf Customers
IP Route Table for VRF "Customers"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.10.10.93/32, ubest/mbest: 2/0, attached
    *via 10.10.10.93, Lo888, [0/0], 01:17:36, local
    *via 10.10.10.93, Lo888, [0/0], 01:17:36, direct
10.10.10.193/32, ubest/mbest: 1/0
    *via 192.168.93.102%default, [200/0], 00:33:51, bgp-65501, internal, tag 65501 (evpn) segid: 900001 tunnelid: 0xc0a85d66 encap: VXLAN

10.88.88.0/24, ubest/mbest: 1/0
    *via 192.168.93.102%default, [200/0], 00:22:37, bgp-65501, internal, tag 65502 (evpn) segid: 900001 tunnelid: 0xc0a85d66 encap: VXLAN

10.93.1.2/31, ubest/mbest: 1/0, attached
    *via 10.93.1.2, Vlan99, [0/0], 00:55:59, direct
10.93.1.2/32, ubest/mbest: 1/0, attached
    *via 10.93.1.2, Vlan99, [0/0], 00:55:59, local
192.168.70.0/26, ubest/mbest: 1/0, attached
    *via 192.168.70.1, Vlan280, [0/0], 00:55:59, direct
192.168.70.1/32, ubest/mbest: 1/0, attached
    *via 192.168.70.1, Vlan280, [0/0], 00:55:59, local
192.168.70.10/32, ubest/mbest: 1/0, attached
    *via 192.168.70.10, Vlan280, [190/0], 00:19:48, hmm

show bgp ipv4 unicast vrf Customers
BGP routing table information for VRF Customers, address family IPv4 Unicast
BGP table version is 35, Local Router ID is 10.0.93.101
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
*>l10.10.10.93/32     0.0.0.0                           100      32768 i
*>i10.10.10.193/32    192.168.93.102                    100          0 i
*>i10.88.88.0/24      192.168.93.102                    100          0 65502 i
* i192.168.70.0/26    192.168.93.102                    100          0 i
*>l                   0.0.0.0                           100      32768 i

show bgp l2vpn evpn vrf Customers
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 37, Local Router ID is 10.0.93.101
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.0.93.101:33047    (L2VNI 28000)
*>l[2]:[0]:[0]:[48]:[b8ac.6f86.b153]:[0]:[0.0.0.0]/216
                      192.168.93.93                     100      32768 i
*>l[2]:[0]:[0]:[48]:[b8ac.6f86.b153]:[32]:[192.168.70.10]/272
                      192.168.93.93                     100      32768 i
*>l[3]:[0]:[32]:[192.168.93.93]/88
                      192.168.93.93                     100      32768 i

Route Distinguisher: 10.0.93.102:3
*>i[5]:[0]:[0]:[24]:[10.88.88.0]/224
                      192.168.93.102                    100          0 65502 i
*>i[5]:[0]:[0]:[26]:[192.168.70.0]/224
                      192.168.93.102                    100          0 i
*>i[5]:[0]:[0]:[32]:[10.10.10.193]/224
                      192.168.93.102                    100          0 i

Route Distinguisher: 10.0.93.101:3    (L3VNI 900001)
*>l[2]:[0]:[0]:[48]:[7c69.f6e0.b057]:[0]:[0.0.0.0]/216
                      192.168.93.93                     100      32768 i
*>l[5]:[0]:[0]:[26]:[192.168.70.0]/224
                      192.168.93.101                    100      32768 i
*>l[5]:[0]:[0]:[32]:[10.10.10.93]/224
                      192.168.93.101                    100      32768 i

show l2route mac-ip all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated (Orp):Orphan
Topology    Mac Address    Host IP         Prod   Flags         Seq No     Next-Hops
----------- -------------- --------------- ------ ---------- ---------------
280         b8ac.6f86.b153 192.168.70.10   HMM    --            0         Local

Outputs from SW2:

sh forwarding vrf Customers

slot  1
=======


IPv4 routes for table Customers/base

------------------+-----------------------------------------+----------------------+-----------------+---------------
--
Prefix            | Next-hop                                | Interface            | Labels          | Partial Instal
l
------------------+-----------------------------------------+----------------------+-----------------+---------------
--
127.0.0.0/8          Drop                                      Null0
10.10.10.93/32       192.168.93.101     nve1
10.10.10.193/32      Receive                                   sup-eth1
*10.88.88.0/24       10.124.124.2                              Ethernet1/10
10.93.1.2/31         Attached                                  Vlan99
10.93.1.2/32         10.93.1.2                                 Vlan99
10.93.1.3/32         Receive                                   sup-eth1
10.124.124.0/30      Attached                                  Ethernet1/10
10.124.124.0/32      Drop                                      Null0
10.124.124.1/32      Receive                                   sup-eth1
10.124.124.2/32      10.124.124.2                              Ethernet1/10
10.124.124.3/32      Attached                                  Ethernet1/10
192.168.70.0/26      Attached                                  Vlan280
192.168.70.0/32      Drop                                      Null0
192.168.70.1/32      Receive                                   sup-eth1
192.168.70.10/32     192.168.70.10                             Vlan280
192.168.70.63/32     Attached                                  Vlan280

sh ip arp suppression-cache local

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface

Ip Address      Age      Mac Address    Vlan Physical-ifindex    Flags

192.168.70.10   00:08:16 b8ac.6f86.b153  280 port-channel93      L

sh ip route vrf Customers
+IP Route Table for VRF "Customers"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.10.10.93/32, ubest/mbest: 1/0
    *via 192.168.93.101%default, [200/0], 00:38:37, bgp-65501, internal, tag 65501 (evpn) segid: 900001 tunnelid: 0xc
0a85d65 encap: VXLAN

10.10.10.193/32, ubest/mbest: 2/0, attached
    *via 10.10.10.193, Lo888, [0/0], 01:22:04, local
    *via 10.10.10.193, Lo888, [0/0], 01:22:04, direct
10.88.88.0/24, ubest/mbest: 1/0
    *via 10.124.124.2, [20/0], 00:27:24, bgp-65501, external, tag 65502
10.93.1.2/31, ubest/mbest: 1/0, attached
    *via 10.93.1.3, Vlan99, [0/0], 01:02:04, direct
10.93.1.3/32, ubest/mbest: 1/0, attached
    *via 10.93.1.3, Vlan99, [0/0], 01:02:04, local
10.124.124.0/30, ubest/mbest: 1/0, attached
    *via 10.124.124.1, Eth1/10, [0/0], 00:27:25, direct
10.124.124.1/32, ubest/mbest: 1/0, attached
    *via 10.124.124.1, Eth1/10, [0/0], 00:27:25, local
192.168.70.0/26, ubest/mbest: 1/0, attached
    *via 192.168.70.1, Vlan280, [0/0], 01:18:54, direct
192.168.70.1/32, ubest/mbest: 1/0, attached
    *via 192.168.70.1, Vlan280, [0/0], 01:18:54, local
192.168.70.10/32, ubest/mbest: 1/0, attached
    *via 192.168.70.10, Vlan280, [190/0], 00:24:34, hmm

show bgp l2vpn evpn vrf Customers
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 12, Local Router ID is 10.0.93.102
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.0.93.101:3
*>i[5]:[0]:[0]:[26]:[192.168.70.0]/224
                      192.168.93.101                    100          0 i
*>i[5]:[0]:[0]:[32]:[10.10.10.93]/224
                      192.168.93.101                    100          0 i

Route Distinguisher: 10.0.93.102:33047    (L2VNI 28000)
*>l[2]:[0]:[0]:[48]:[b8ac.6f86.b153]:[0]:[0.0.0.0]/216
                      192.168.93.93                     100      32768 i
*>l[2]:[0]:[0]:[48]:[b8ac.6f86.b153]:[32]:[192.168.70.10]/272
                      192.168.93.93                     100      32768 i
*>l[3]:[0]:[32]:[192.168.93.93]/88
                      192.168.93.93                     100      32768 i

Route Distinguisher: 10.0.93.102:3    (L3VNI 900001)
*>l[2]:[0]:[0]:[48]:[7426.ac76.500b]:[0]:[0.0.0.0]/216
                      192.168.93.93                     100      32768 i
*>l[5]:[0]:[0]:[24]:[10.88.88.0]/224
                      192.168.93.102                                 0 65502 i
*>l[5]:[0]:[0]:[26]:[192.168.70.0]/224
                      192.168.93.102                    100      32768 i
*>l[5]:[0]:[0]:[32]:[10.10.10.193]/224
                      192.168.93.102                    100      32768 i

show l2route mac-ip all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated (Orp):Orphan
Topology    Mac Address    Host IP         Prod   Flags         Seq No     Next-Hops
----------- -------------- --------------- ------ ---------- ---------------
280         b8ac.6f86.b153 192.168.70.10   HMM    --            0         Local

f00z · ‎05-11-2020

I have a lab set up with several gen1 (trident2) 9300's , I'll set up this exact scenario sometime soon and get back to you. Src mac will be the rmac of the encapsulating switch when it's routed, that's normal for anything. TTL expired means it's routing loop, so probably sw2 not getting the arp(and therefore the type 2 route /32 inserted locally) so it's sending it back to sw1 which is sending it back to sw2 which is sending it back to sw1 etc... do a traceroute instead of ping I bet that is what you will see... well that would be with pinging a client on sw2 but u are pinging loopback, so the loopback must not be inserted into the table.. I keep thinking you are trying to ping a client on sw2 not the loopback haha. So the ingress packet shouldn't loop it should terminate at the loopback, but the egress packet would loop if there was a conflicting route back to the client. Loopback wouldn't have an arp entry but would have a corresponding type 5 entry from redistribution or the network statement which should be fine. We know this works in non VPC environment. I will test with VPC when I get a chance the same setup, no reason why it shouldn't work and no reason it should give u a ttl expiry unless there's a routing loop, do a traceroute and see where it is looping.

ez9 · ‎05-11-2020

OK. Thank you for your answer. I tried a traceroute from clien1 to loopback but I am seeing the default gateway IP (192.168.70.1) in every hop. I tried to capture the trace packets and I can see that arriving to SW2. I can't see if the SW2 generates an answer.

Thank you again for your time. I am awaiting for your results.