Cisco Nexus Routing Lookup Clarification

[john.gregory]

My setup is Spine-Leaf architecture using Nexus 9300 running on NX-OS v10.3.5. I have a scenario wherein I am doing a traceroute between two different subnets across two DC (DC1 and DC2). 

The DC-1 ESXi Host gateway IP is a distributed anycast gateway hosted in Service Leaf01 & Leaf02. To reach the DC-2 ESXi Host, the traffic will go through the Border Leaf, the border leaf has a default route to Fortigate Firewall via vlan 609. The Firewall has the specific route to reach DC-2 ESXi Host via DC-2 Core Switch via vlan 602.

The border leaf has a specific static route to backup server subnet in DC-2 Core Switch. The purpose of this route is to bypass Fortigate Firewall for backup replication.

See attached logical diagram for reference.

I encounter a strange behavior wherein if I traceroute from ESXi Host in DC1 to ESXi Host in DC2, once the packet reaches the border leaf, instead of sending the packet to the next-hop IP (FW) as per routing table, for unknown reason it will have an additional hops within the Border Leaf that shows SVI 603 then shift to vlan 609 before sending the packet to the Firewall. See hop no.2 in the attached diagram. Not sure exactly how to fix it and I don't clearly understand why the route lookup happens this way.

[john.gregory]

I read through again this document https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/103x/configuration/vxlan/cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-release-103x/m_configuring_vxlan_93x.html, and now I think its a normal behavior based on the statement below. Can someone help to confirm my thoughts on this?

• For traceroute through a VXLAN fabric when using L3VNI, the following scenario is the expected behavior:

  If L3VNI is associated with a VRF and an SVI, the associated SVI does not have an L3 address that is configured but instead has the "ip forward" configuration command. Due to this interface setup it cannot respond back to the traceroute with its own SVI address. Instead, when a traceroute involving the L3VNI is run through the fabric, the IP address reported will be the lowest IP address of an SVI that belongs to the corresponding tenant VRF.