Peer Gateway is not working when SVI is missing.

[James Tu]

Hi All,

I have a lab topology like this but loop avoidance seems not to work and the traffic is duplicated on N9K-2 when the peer gateway is enabled.

Interconnections.

- Server1 connects to Router 1.
- Routers use a subinterface to connect to N3Ks.
- N3Ks are not forming VPC but work as extensions for the Router.
- N93180-1 and N93180-2 form VPC and virtual port channels with N3K.
- N93180-2 acts like a simple layer 2 switch and uses a port channel to connect with N93180-1&N93180-2.

-----------------------------------------------------------------------------------------------------------Layer2

Only N3K has trunk allow vlans setting like below.

N93180-1 and N93180-2 allow all vlans on all interfaces.

------------------------------------------------------------------------------------------------------------

Layer3

Router1 has sub-interfaces with vlan tags, 10,12,18. Subinterfaces with tags 10 and 13 are in the VRF.
Router2 has sub-interfaces with vlan tags, 11,13,19. Subinterfaces with tags 11 and 12 are in the VRF.

N93180-1 has SVIs 10,13,18. SVI 10 and 13 are in VRF.
N93180-2 has SVIs 11,12,19. SVI 11 and 12 are in VRF.

Router1 and Router2 both establish OSPF relationships with N93180-1 and N93180-2 via the three sub-interfaces.

Also, Router1 and Router2 both have sub-interfaces, 100 in Global and 101 in VRF, and establish OSPF neighbors with each other.

Router 1 connects to Server 1 using an interface in Global VRF.
N93180-1 and N93180-2 act as a gateway for Server2 with HSRP configured.
The Subnet of Server2 is configured in VRF so Router1 and Router2 leak it from VRF to Global.

------------------------------------------------------------------------------------------------------------

I run iperf3 from server1 to server2. The maximum bandwidth is 10G.

If the peer gateway is enabled and the traffic path is Server1 -> Router 1 -> N3K-1 -> N93180-2, the traffic would be sent out to N3K-2, N93180-3, and also N93180-1 via the peer link.

According to a document about loop avoidance, the traffic should not send to N93180-3 by N93180-1 but it does.
So, I can see almost 10G traffic from both links towards N93180-1(e1/1) and N93180-2(e1/3).

If I disable the peer-gateway feature, the problem is solved.
N93180-2 is not sent traffic out to N3K-2 and N93180-3 but only N93180-1.
(Real-time monitoring on Cacti)

Here are vpc settings

N93180-1

vpc domain 10
  peer-switch
  role priority 10
  system-mac ce:cd:be:ef:ab:cd
  peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf keepalive
  peer-gateway
  layer3 peer-router
  ip arp synchronize

interface Vlan10
  no shutdown
  vrf member VRF
  ip address 192.168.10.253/24
  ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  hsrp 10
    ip 192.168.10.254
    preempt

N93180-1
vpc domain 10
  peer-switch
  role priority 20
  system-mac ce:cd:be:ef:ab:cd
  peer-keepalive destination 10.1.1.1 source 10.1.1.2 vrf keepalive
  peer-gateway
  layer3 peer-router
  ip arp synchronize

interface Vlan10
  no shutdown
  vrf member VRF
  ip address 192.168.10.253/24
  ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  hsrp 10
  ip 192.168.10.254
  preempt
---

As I know, a peer gateway is used to forward the traffic to the destination Mac belonging to the vpc peer member and loop avoidance should kick in when traffic is received from the peer link and it's about to send out to another vpc member ports.

Best regards,
James

[James Tu]

Hi All,

I have some features related to VXLAN enabled on both N93180 and after I disable them, no traffic reaches Server2.

The feature I disabled.

feature fabric forwarding
feature nv overlay
nv overlay evpn

After capturing some packets, I found that the peer gateway does not work as expected.
When traffic reaches N93180-2, it forwards it via a peer link to N93180-1.
Due to the loop avoidance, the traffic does not send out to N93180-3.

After a couple of hours of tests, I created an SVI of VLAN 10 in VRF and assign an IP address to it on N93180-2, it starts to forward traffic to the downlink. No traffic is sent to the peer link now.
Without IP assigned, N93180-2 still forwards traffic to N93180-1.
I also tried another thing, the IP address on N93180-2 does not have to be in the same subnet as the SVI of N93180-1.

Then I enabled the VXLAN feature back to both N93180 and using anycast gateway to replace HSRP.
No duplicated traffic shows anymore.
Everything works as expected.

----------------------------------------------------------------------------------------------------------

My conclusion.

For the peer gateway feature to work properly, both switches running as VPC should have the same SVI configured.
The IP address should be configured but no need to be in the same subnet.

---------------------------------------------------------------------------------------------------------
I can't find any documents talking about this.
Is my design a valid design?

My theory is that SVI is only enabled on one switch so the other one would consider it a pure layer 2 switch for that VLAN.
Therefore, it would not help forward the traffic even if the peer gateway feature is enabled.
As for why an IP address must be configured on SVI, I still don't know why.

Thank you.

Best regards,
James