L3 out interface with MTU 9216 wont send packets larger the 1500 bytes

doridor · ‎06-27-2023

I am trying to increase the MTU of my L3out routed ports to 9216,

when I increased the MTU on one of the ports and pinged it form the other side with a 9216 byte icmp packet, it received the request and responded with a 1500 byte icmp reply. And once I try to initiate a ping larger then 1500 byes from the interface it doen't send anything.

When running "show interface {interface}" the MTU is set to 9216.

Does anyone have any idea why this is happening?

r.heitmann · ‎06-28-2023

is the device (router) you try to reach via ping from ACI is directly connected or is there a switch in between?

is this switch enabled for jumbo-MTU, too?

...and -

is the device you try to reach itself enabled for jumbo-MTU?

doridor · ‎06-28-2023

The problem isn’t with the device i am trying to reach. The routed port on the border leaf will not even sent the packets if they are larger then 1500 and the df flag is set

r.heitmann · ‎07-04-2023

I guess, you're using the "iping"-command beeing connected using SSH to a ACI-Leaf switch.

-V = <TENANT>:<VRF>
-F = don't fragment
-s = packet-size

The packet-size-parameter of "iping" specifies the "data-portion" of the ICMP-Message sent.

like a windows-ping would do
unlike a cisco-ios-router would do

So, if IP-MTU=9000 should be possible specify 8972 bytes.

20 Bytes for IPv4-Header
8 Bytes for ICMP-Header

get added.

The result is an IP-Packet with size 8972+20+8=9000 Byte.

In my example, an IP-MTU of 9000 is possible, so

"-s 8972" => works
"-s 8973" => fails

leaf1# iping -V MTUTEST:DEFAULT 172.16.0.202 -F -s 8972
PING 172.16.0.202 (172.16.0.202) from 172.16.0.201: 8972 data bytes
8980 bytes from 172.16.0.202: icmp_seq=0 ttl=254 time=1.003 ms
8980 bytes from 172.16.0.202: icmp_seq=1 ttl=254 time=6.615 ms
8980 bytes from 172.16.0.202: icmp_seq=2 ttl=254 time=0.864 ms
8980 bytes from 172.16.0.202: icmp_seq=3 ttl=254 time=0.893 ms
^C
--- 172.16.0.202 ping statistics ---
4 packets transmitted, 4 packets received, 0.00% packet loss
round-trip min/avg/max = 0.864/2.343/6.615 ms

leaf1# iping -V MTUTEST:DEFAULT 172.16.0.202 -F -s 8973
PING 172.16.0.202 (172.16.0.202) from 172.16.0.201: 8973 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
^C
--- 172.16.0.202 ping statistics ---
4 packets transmitted, 0 packets received, 100.00% packet loss

just to prove, that the packet gets lost by MTU too big - I'll repeat the command without the "don't fragment"-option:

now the "too big" packet gets fragmented inside the leaf making the test a success

leaf1# iping -V MTUTEST:DEFAULT 172.16.0.202 -s 8973
PING 172.16.0.202 (172.16.0.202) from 172.16.0.201: 8973 data bytes
8981 bytes from 172.16.0.202: icmp_seq=0 ttl=254 time=1.084 ms
8981 bytes from 172.16.0.202: icmp_seq=1 ttl=254 time=0.893 ms
8981 bytes from 172.16.0.202: icmp_seq=2 ttl=254 time=1.014 ms
8981 bytes from 172.16.0.202: icmp_seq=3 ttl=254 time=1.093 ms
8981 bytes from 172.16.0.202: icmp_seq=4 ttl=254 time=1.034 ms
^C
--- 172.16.0.202 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.893/1.023/1.093 ms

Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

r.heitmann · ‎07-04-2023

if this doesn't help we might need to check your L3Out-setup if your neighbor-device is fine.

...some ideas

verify the MTU of the L2-Interface your L3-Out uses (indirectly)
- 9216 Bytes as in your case
- this is the upper limit for Ethernet-Frames

leaf1# show interface ethernet 1/48
Ethernet1/48 is up
admin state is up, Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 0000.0000.0000 (bia 7c0e.cecb.cc0a)
  MTU 9216 bytes, BW 1000000 Kbit, DLY 1 usec
...

For L3-IP traffic there might be a (lower) limit at the IP-Interface.

find out the IP-Interface of your L3-Out

leaf1# show ip int brief vrf MTUTEST:DEFAULT
IP Interface Status for VRF "MTUTEST:DEFAULT"(8)
Interface            Address              Interface Status
eth1/48.29           172.16.0.201/30      protocol-up/link-up/admin-up
vlan19               172.16.100.254/24    protocol-up/link-up/admin-up
lo2                  172.16.255.201/32    protocol-up/link-up/admin-up

find out the IP-MTU of your L3-Out
- 9000 Bytes in my case
- that's the real limit for IP

leaf1# show int e1/48.29
Ethernet1/48.29 is up
admin state is up, Dedicated Interface, [parent interface is Ethernet1/48
  Hardware: 1000/10000 Ethernet, address: 0000.0000.0000 (bia 7c0e.cecb.cc0a)
  Internet Address is 172.16.0.201/30
  MTU 9000 bytes, BW 1000000 Kbit, DLY 1 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast

Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

wrgu · ‎11-24-2023

Not sure if you already know the reason.

Anyway, looks like the iping mtu is configured by the control plane MTU which is configured under the system setting. It took me awhile to figure it out.