Nexus doesn't pass VxLAN packet to Next-hop

[terry_lin1]

Hi All

I've configured two Nexus 9Kv in my EVE-NG lab, and I check the configuration and debug some packet process on the 9Kv. It's doesn't see any VxLAN packet with src 5000.0003.0000(left side FG port1) and dst 5000.0004.0000(right side FG port1) out Eth1/1, when left FG send ethernet frame to right FG. In addition, from the 9K1 debug output, it looks like has processes recorded. 

[ADP_89]

Hello,

Your configuration looks right for a static ingress replication. I actually tried on my GNS3 but on NX version 7.0(3)I7(3) and I can ping from one side to the other. I have used an ipterm docker image to simulate the 2 hosts behind Eth1/2.

It might be something wrong with NXosV 9, have you tried to use a different image?

ADP

[terry_lin1]

Hi Sir

Yes, it since all right on VxLAN and MAC<->VETP learned, just unicast frames(5000.0003.0000<->5000.0004.0000) between FG1 and FG2 cannot communicate. Before I posted this problem, I thought the unicast Ethernet frame (no other layer header or data) cannot pass N9Kv. However, a broadcast frame looks ok for this scenarios. I've tried 7.0(3).I7.5 and 7.0(3).I7.3, 9.2(1) on eve-ng. all of the above version show the same result. Thank you  

[ADP_89]

Can you take a capture with Wireshark over GNS3 of those packets and upload it here? So I can see what traffic is that and replicate it via scapy.

Thanks,

ADP

[terry_lin1]

Hi Sir

here is the packet capture from N9K1 and N9K2, I also posted the few pictures of devices capture. 5000.0004.0000 is FG1's port1 MAC and 5000.0003.000 is FG2's port1 MAC. The packet passed VxLAN only broadcast from 04 or 03, none of the unicast frames with src 04 to 03, it looks like broke on N9K1. The strange thing is N9K1 has processed the unicast frames with src 04 to 03 from the deb l2fwder packet output " Out vlan 10002 BD 2 XXXXXXX...... nve-peer1.1.1.2 (nve-peer1)".

[ADP_89]

Hello,

So I have done some hacking around this. The problem in your packets resides on the Ethertype which is set to 0x0843, with scapy I have edited the packet and send it with an ethertype of 0x0800 and I saw it reaching the other side. To be honest I think this is a Nexus9000V limitation as VXLAN should tunnel any Ethernet frame. As far I can see NX9000v only encapsulates ARP,IPv4 and IPv6 packets.

I have then tested into a real lab with a couple of NX9300, and it worked. Even the Ethertype 0x0843 reached the other side.

Cheers,

ADP

[terry_lin1]

Hi Sir

Big thank you for your reply and time for this scenario, if real NX9300 can transport Ethertype 0x0843/0x0844, it can be sure is N9000v limitation. May I ask another help that posts the pcap or picture for real device passed  Ethertype 0x0843/0x0844 from you if it doesn't take you much time? Thanks again. 

[ADP_89 · Accepted solution]

Hello,

I can try as at the moment I am not phisically where the lab is and I won't be there for the next 2 weeks. I will try to do an erspan over the current lab cabling.

Please remember to mark this topic as solved and mark all the usefull replies.

Thanks,

ADP