This is a blog about configuring jumbo frames in OpenStack in order to scale neutron and nova in a production datacenter using Cisco's UCS servers, UCS Fabric Interconnects and Nexus 9000.
In computer networking, MTU (Maximum Transmission Unit) is the size (in bytes) of the largest packet that can be transferred by an interface without IP fragmentation.
Fixed before transmitting data (Ethernet)
Negotiated during handshake/connect time (point-to-point serial links)
Dynamically determined on-the-fly while transmitting data
The default MTU is 1500 bytes. Jumbo MTU is 9000 bytes.
Here is an IP packet.
IP fragmentation is an process that breaks datagrams into smaller pieces (fragments), so that packets may be formed that can pass through a link with a smaller maximum transmission unit (MTU) than the original datagram size. The fragments are reassembled by the receiving host. IP fragmentation adds additional processing overhead on the interface.
IP fragmentation is controlled by the DF (Don't Fragment) bit.
IP Fragmentation (DF bit in IP header):
Here are the neutron configurations needed to enable jumbo MTU (9000 bytes) in OpenStack.
9008 bytes from 10.23.223.21: icmp_seq=0 ttl=254 time=1.384 ms
9008 bytes from 10.23.223.21: icmp_seq=1 ttl=254 time=0.993 ms
9008 bytes from 10.23.223.21: icmp_seq=2 ttl=254 time=0.919 ms
9008 bytes from 10.23.223.21: icmp_seq=3 ttl=254 time=0.927 ms
9008 bytes from 10.23.223.21: icmp_seq=4 ttl=254 time=1.002 ms
--- 10.23.223.21 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.919/1.044/1.384 ms
Jumbo packet (with MTU 9000 bytes) captured in Wireshark packet sniffer:
Jumbo frames in OpenStack with neutron SR-IOV ports:
I observed that nova VMs attached to a neutron SR-IOV port will also pass jumbo MTU packets. Below is how jumbo MTU is seen on the interface of a nova VM attached to a neutron SR-IOV port.
[cloud-user@sr-iov-vm ~]$ ip a | grep 9000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
Path MTU Discovery:
Path MTU Discovery (PMTUD) is a technique to determining the MTU of the network path between source and destination. PMTUD works by setting the Don't Fragment (DF) flag bit in the IP headers of outgoing packets. Then, any device along the path whose MTU is smaller than the packet will drop it, and send back an Internet Control Message Protocol (ICMP) Fragmentation Needed (Type 3, Code 4) message containing its MTU, allowing the source host to reduce its Path MTU appropriately. This process is repeated until the MTU is small enough to traverse the entire path without fragmentation.
Advantages of jumbo frames:
Lesser packet drop
Interfaces process fewer packets per second
Disadvantages of jumbo frames:
Slower per-packet processing since each packet is big (9000 byes)
Bigger packets on the wire may increase lag and latency
Packet corruption causes huge retransmits and congestion