Nexus 9300 tunnel feature fragmentation problem

ewaizel · ‎03-07-2019

Hi everybody!

Hope any of you had the same problem.

We are using the tunnel feature in NX-OS. When the packets are bigger than the tunnel MTU and fragmentation is needed, the IP fragments created by the switch have different ID at an IP level and cannot be reassembled by receiving node.

We have multiple Nexus 93180YC-EX in different locations and with different versions: 7.0(3)I7(3) & 9.2(2).

They all seem to be having the problem.

This is my test environment:

In NX-OS:

feature tunnel

interface Tunnel2
ip address 172.16.1.1/30
tunnel source Vlan212
tunnel destination 172.23.1.1

mtu 1000
no shutdown

ip route 1.1.1.1/32 Tunnel2

I then go to a L2 access switch routing through the previous NX-OS switch
Access Switch#ping 1.1.1.1 size 2000 repeat 1

The tunnel in this test terminates at a server running Wireshark (IP 172.23.1.1)

I can see the capture with 3 packets not necessarily in order. The 1st and 2nd packets have IP offset of 0 & 976; the ID is 0x80f0.

The last packet has an IP offset of 1480; the ID is 0x00f0!!!

One bit is wrong! instead of 0x80f0 it is 0x00f0. The 3 fragments cannot be reassembled by the end recipient.

Any ideas or similar experiences?

Andrea Testino · ‎03-11-2019

I've tried the following topology & configuration below and I'm not seeing my IP IDs or Fragment Offsets differ between each hop:

Here are the ICMP Requests from N9K-3 to N9K-5 as they get fragmented:

############################
#### SENDING N9K, N9K-3 ####
############################

These come out fragmented themselves as I'm sending with a size of 4000 and the SVI here has default MTU (1500):

N9K-3# ethanalyzer local interface inband display-filter ip limit-c 0

Capturing on inband
2019-03-11 13:29:32.354015   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=0, ID=ee69)
2019-03-11 13:29:32.354104   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=1480, ID=ee69)
2019-03-11 13:29:32.354160   172.23.2.1 -> 1.1.1.1      ICMP Echo (ping) request


##########################
##### FRAGMENTING N9K ####
##########################

N9K-4# ethanalyzer local interface inband display-filter ip limit-c 0

Original packet arriving fragmented as sent by N9K-3:

Capturing on inband
2019-03-11 18:59:31.678237   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=0, ID=ee69)
2019-03-11 18:59:31.678309   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=1480, ID=ee69)
2019-03-11 18:59:31.678344   172.23.2.1 -> 1.1.1.1      ICMP Echo (ping) request

Fragmenting further in order to send out of Tunnel2 (MTU of 1000):

2019-03-11 18:59:31.678991   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=0, ID=ee69)
2019-03-11 18:59:31.679228   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=976, ID=ee69)
2019-03-11 18:59:31.679298   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=1480, ID=ee69)
2019-03-11 18:59:31.679364   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=2456, ID=ee69)
2019-03-11 18:59:31.679446   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=2960, ID=ee69)
2019-03-11 18:59:31.679529   172.23.2.1 -> 1.1.1.1      ICMP Echo (ping) request

##########################
##### RECEIVING N9K-5 ####
##########################

N9K-5# ethanalyzer local interface inband display-filter ip limit-c 0

Capturing on inband
2019-03-11 18:05:30.499740   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=0, ID=ee69)
2019-03-11 18:05:30.499925   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=976, ID=ee69)
2019-03-11 18:05:30.500004   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=1480, ID=ee69)
2019-03-11 18:05:30.500056   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=2456, ID=ee69)
2019-03-11 18:05:30.500153   172.23.2.1 -> 1.1.1.1      IP Fragmented IP protocol (proto=ICMP 0x01, off=2960, ID=ee69)
2019-03-11 18:05:30.500214   172.23.2.1 -> 1.1.1.1      ICMP Echo (ping) request

Notice all offsets above line up with the second half of the "fragmenting N9Ks" output.

I'll unicast you to get specific configurations / setup in the event something differs and I can gladly test again -- We can circle back to the forum with the "answer" at the end.

Thank you,

- Andrea, CCIE #56739 R&S