MTU without configuring IP MTU on Cat9k

[j.a.m.e.s]

There are many articles explaining "MTU" and "IP MTU", but I can't work out what would be the impact of setting interface/system MTU, but leaving ip mtu unconfigured. My goal is jumbos with the minimum config.

system mtu 9198

I just want that end hosts can pass jumbo at layer2 and layer3 and that OSPF will establish with the upstream router, which would be configured in the same way. It would be really painful go through every SVI and Interface with an IP address to set ip mtu.

The Cat9k MTU Troubleshooting guide doesn't really say what happens if you leave the IP MTU at default.

[j.a.m.e.s]

Here's what happens when I set only system mtu 9198:

cat9300-01# show vlan mtu

VLAN    SVI_MTU    MinMTU(port)      MaxMTU(port)     MTU_Mismatch
---- ------------- ----------------  ---------------  ------------
1    9198          9198              9198              No <-- Looks good
2    9198          9198              9198              No <-- Looks good
50   9198          9198              9198              No <-- Looks good
202  9198          9198              9198              No <-- Looks good
cat9300-01#show inter te1/1/1 | in MTU
 MTU 9198 bytes, BW 1000000 Kbit/sec, DLY 10 usec, <-- Looks good

Under the interface I didn't set "mtu" or "ip mtu". Does anyone know if this was sufficient for end hosts to send jumbos and OSPF to consider the interface as MTU 9198 please? 

[balaji.bandi]

Depending on the requirement, if the SD-Access deployment VXLAN needs more, Jumbo frames are required for Cat 9K Models.

Here is a good example to give you direction :

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/217233-troubleshoot-mtu-on-catalyst-9000-series.html

OSPF required both the side to be same, if not you can configure MTU ignore command -

The interface command ip ospf mtu-ignore disables this check of the MTU value in the OSPF DBD packets. Thus, use of this command allows the OSPF adjacency to reach the FULL state even though there is an interface MTU mismatch between two OSPF routers.

[j.a.m.e.s]

@balaji.bandi Agreed, the MTUs at each side of the link must match for OSPF (or the ignore command must be in place), but thankfully setting "ip mtu" is unnecessary ("system mtu" or "mtu" is enough)

[balaji.bandi]

It depends on the requirement; if not the interface level, then inherited settings will be effective.

[Flavio Miranda]

@j.a.m.e.s 

if you set system MTU alone, this will be applied to every interface. If you want to change this, set the MTU on the interface individually.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-3/configuration_guide/int_hw/b_173_int_and_hw_9300_cg/configuring_system_mtu.html

[j.a.m.e.s]

Thank you, my question though is really about the confusion with so many commands to amend MTU. Why is there an "ip mtu" setting? Are we saying this is a way to modify MTU of packets with an IP header while still keeping non-IP packets locked below 1500bytes? It seems a bit niche these days as almost everything is IP.

[MHM Cisco World]

Cases

A-

Client1 vlanx to cleint2 vlanx 

Both clients use 9000 

Here traffic is bridge and hence you need to modify l2 frame size by ""MTU""

B- Client1 vlanx  client2 vlany 

Only client1 use jumbo frame

Here since traffic is routing then we need to modify l3 packet by using ""ip mtu""

Also

Ospf need to match l3 packet ""ip mtu""

[j.a.m.e.s]

@MHM Cisco World I originally thought that "ip mtu" was needed for OSPF and L3 Jumbo, but it seems to be no longer the case (thankfully):

# Cat9k:
interface TenGigabitEthernet1/1/6
 no switchport
 mtu 9198
 ip address 10.0.184.69 255.255.255.254
  ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 0 xxxxxx
 ip ospf network point-to-point
 ip ospf 1 area 0

# ISR4k:
interface GigabitEthernet0/0/0
 mtu 9198
 ip address 10.0.184.68 255.255.255.254
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 xxxxxxx
 ip ospf network point-to-point
 ip ospf 1 area 0.0.0.0

OSPF is up and stable with routes exchanged and ping across the L3 link with large pkts is ok:

isr4k#sho ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
10.0.184.1 0 FULL/ - 00:00:33 10.0.184.69 GigabitEthernet0/0/0

isr4k#ping 10.1.184.66 size 9000 df-bit <- N9k physical L3 interface across Cat9k then carrier fibre circuit
Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to 10.1.184.66, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!

I didn't set "ip mtu" anywhere and jumbos are flowing across L3 interfaces at 9000 bytes, OSPF is stable.

Preview

[Joseph W. Doherty]

The difference between MTU and IP MTU is the former adjusts the physical frame size allowed, for any/all traffic, while the latter adjusts MTU for only IP traffic.

In theory, MTU is always the maximum capable frame size, by standard, for a particular medium.  With either MTU or IP MTU we're  possibly "spoofing" a reduced MTU, for various reasons.

In the forgoing, "reduced" might seem contrary to when adjusting MTU for jumbo Ethernet.  However, jumbo Ethernet frame size is both non-standard and larger than standard, so when we have a jumbo Ethernet interface with a MTU of 1500, we've reduced the actual frame size allowed to what would be the standard MTU, but it's not really the NIC's MTU.

An interesting side note, that a Cisco engineer relayed to me (I've not verified it), was Cisco network device jumbo Ethernet NICs, even when set with a MTU set to 1500 will accept frames up to their (actual/physical) MTU, but such an interface shouldn't transmit a frame larger than its MTU setting (even if physical capable).  (Personally, if this is true, I consider it poor behavior, as I believe the NIC should operate as if it truly was not jumbo capable, i.e. not accept a frame larger than it's configured MTU.)

[j.a.m.e.s]

Thank you Joseph, that provides some more useful info.

It does seem a bit niche that someone would want different MTUs for IP and Non-IP packets, but I think I've established that for the vast majority of use cases on modern hardware and software on  Cat9k/ISR4k/N9k we can pretend the "ip mtu" command doesn't exist and stick with mtu or system mtu.