MPLS - MTU and packet loss with payload greater than 1492 bytes

[DarrenB_UK]

Hello all,

The following topology has been deployed using Catalyst 9300 switches at the core and ASR920-12SZ-IM routers for WAN aggregation.  OSPF is the IGP with MP-BGP configured for MPLS L3VPNs to support multiple customers.  All interfaces are L3 routed ports.

The system MTU on the C9300s is set at 9168 with the interface MTU on the ASR920 set the same.  IP MTU is configured as 1500.  Usually, the MPLS MTU is derived from the system/interface MTU and indeed, a show run all on the interfaces shows the following:

DC2-C9300-01#show run all | section interface TenGigabitEthernet1/1/2
::::
ipv6 mfib forwarding input
ipv6 mfib forwarding output
ipv6 mfib cef input
ipv6 mfib cef output
mpls mtu 9168
mpls ldp igp autoconfig
mpls ldp igp sync
snmp trap link-status
::::

Additionally, a show mpls interfaces detail on the C9300 also shows the MTU of 9168 on all interfaces.

DC2-C9300-01#show mpls interfaces detail | include MTU
        MTU = 9168
        MTU = 9168
        MTU = 9168
        MTU = 9168
        MTU = 9168
        MTU = 9168

And on the ASR920

DC2-ASR920-01#show mpls interfaces detail | include MTU
        MTU = 9168
        MTU = 9168
        MTU = 9168

Despite this, we are experiencing packet loss (from an end device connected to a CE) when the IP packet size is greater than 1492 bytes with df-bit set.  A consequence of this is that certain applications are failing since the TCP segment size will still negotiate at 1460.  I do not want to use the ip tcp adjust-mss command as this is a fudge!

We are using Cisco's recommended release for all devices and the config guide for MPLS on the C9300 states:

Now this seems to contradict what the show mpls interface detail and the show run all on the interface reveals.  Is there a way to verify the true MPLS MTU size on C9300 switches as there is on the ASR920?

The following output clearly shows that despite the MPLS MTU showing as 9168 in the aforementioned commands, it is in fact, only 1504 bytes.

DC2-ASR920-01#show platform hardware pp active feature mpls mtu-table 
MPLS MTU Table

Index    MTU  Ref-Count   
------------------------
0       1504  1           
1       1604  1           
2          0  0           
3          0  0           
4          0  0           
5          0  0           
6          0  0           
7          0  0  

This is what the config guide for the ASR920 states about MPLS MTU:

If it transpires that the MPLS MTU does indeed track the IP MTU, what are the consequences of raising the IP MTU on all core devices to say 2000 to enable the MPLS MTU to allow larger packet sizes?

Many thanks in advance.

[Francesco Molino]

Hi

On release 16.12, mpls isn’t supported.
These catalyst 9300 are the core devices you’re talking about? Max ip mtu will be 1500.
I would use the path MTU discovery to avoid any performance issues due to fragmentation.

[DarrenB_UK]

Thanks for the reply Francesco.  MPLS is indeed supported on the Catalyst 9300 platform with a few limitations.

Namely the inability to manually configure the MPLS MTU command under an interface.  I've included below, configuration from the switch.

!---System MTU value

DC2-C9300-01#show system mtu
Global Ethernet MTU is 9168 bytes.

!---MPLS enabled interfaces

DC2-C9300-01#show mpls interfaces 
Interface IP Tunnel BGP Static Operational
GigabitEthernet1/0/23 Yes (ldp) No No No Yes 
Te1/1/1 Yes (ldp) No No No Yes 
Te1/1/2 Yes (ldp) No No No Yes 
Te1/1/7 Yes (ldp) No No No Yes 
Te1/1/8 Yes (ldp) No No No Yes

!---MPLS interface detail

DC2-C9300-01#show mpls interfaces tenGigabitEthernet 1/1/2 detail 
Interface Te1/1/2:
Type Unknown
IP labeling enabled (ldp) :
IGP config
LSP Tunnel labeling not enabled
IP FRR labeling not enabled
BGP labeling not enabled
MPLS operational
MTU = 9168

!---Standard running config for example interface

DC2-C9300-01#show run interface tenGigabitEthernet 1/1/2
Building configuration...
Current configuration : 295 bytes
!
interface TenGigabitEthernet1/1/2
description *** DC2-ASR920-01 - Te0/0/12 (LSP) ***
no switchport
ip address 10.88.255.9 255.255.255.252
no ip redirects
ip mtu 1500
ip ospf message-digest-key 1 md5 *******
ip ospf network point-to-point
ip ospf 64532 area 0
end

!---Hidden MPLS MTU config (default value)

DC2-C9300-01#show run all | section interface TenGigabitEthernet1/1/2
::::
ipv6 mfib forwarding input
ipv6 mfib forwarding output
ipv6 mfib cef input
ipv6 mfib cef output
mpls mtu 9168
mpls ldp igp autoconfig
mpls ldp igp sync
snmp trap link-status
cts role-based enforcement
no mka pre-shared-key
mka default-policy
autonomic
arp arpa
arp timeout 14400
spanning-tree port-priority 128
spanning-tree cost 0
::::

Referring back to the excerpt from the configuration guide for 16.12:

It does state that MPLS MTU will be the same as the IP MTU so despite what the verification command output displays, it should actually be 1500 since this is clearly configured under the interface config.  I am concerned that Cisco's own "show" commands suggest otherwise.

We simply want to allow end-to-end unfragmented packets over the core so want to avoid PMTUD.  May have to open a case with TAC to establish if there is any specific ASIC verification command that will reveal the true MPLS MTU value used on the interface!

Cheers,

[DarrenB_UK]

Just a quick update on this following configuration on the ASR920.  I have also raised a case with TAC to establish exactly how the Catalyst 9300 platform tracks MPLS MTU despite glaring confusion by the show commands.

!---MPLS MTU reported by standard command

DC2-ASR920-01#show mpls interfaces tenGigabitEthernet 0/0/12 detail
Interface Te0/0/12:
Type Unknown
IP labeling enabled (ldp) :
Interface config
IGP config
LSP Tunnel labeling not enabled
IP FRR labeling not enabled
BGP labeling not enabled
MPLS operational
MTU = 9168

!---MPLS MTU reported by the platform

DC2-ASR920-01#show platform hardware pp active feature mpls mtu-table 
MPLS MTU Table
Index MTU Ref-Count 
------------------------
0 1504 1 
1 1604 1 
2 0 0 
3 0 0 
4 0 0 
5 0 0 
6 0 0 
7 0 0

!---Command to enable the configuration of MPLS MTU (which now tracks the interface MTU if not configured explicitly)

DC2-ASR920-01(config)#platform mpls mtu-enable

!---MPLS MTU now reported by the platform

DC2-ASR920-01#show platform hardware pp active feature mpls mtu-table 
MPLS MTU Table
Index MTU Ref-Count 
------------------------
0 1504 1 
1 1604 1 
2 9172 3 
3 0 0 
4 0 0 
5 0 0 
6 0 0 
7 0 0

Any help would be appreciated!

[Giuseppe Larosa]

Hello @DarrenB_UK ,

increasing the IP MTU size on the core links looks like the only option to avoid MTU issues.

Your choice of core devices in Catalyst 9300 look like uncommon, however if these are the available devices you can only try to adapt to their limitations.

Hope to help

Giuseppe

[DarrenB_UK]

Thanks Giuseppe,

The choice of hardware was both commercial and functional.  Essentially, the switches are PE devices for multi-tenant internal  networks in a private cloud.  We extensively researched the feature support and the only functional stretch was the MPLS requirement using MP-BGP.  Other than providing directly connected networks to be redistributed into MPLS L3VPNs (VRFs), we will need to configure two of the devices as MPLS ASBRs using inter-AS option B, which is supported.

Increasing the IP MTU to 2000 looks like a viable option as we have one requirement for a customer payload of 1600 bytes.  So the MPLS MTU (if it does track the IP MTU), will be correctly sized to accommodate the labelled packets.

Still surprised that the show mpls interfaces detail and the show run all | section interface clearly shows the MPLS MTU as 9168! 

Cheers,

Darren

[Francesco Molino]

On these platforms, MPLS MTU won't work like it should on ASR devices that are more convenient for MPLS core.
Anyways, as @giuseppe said, you have to increase the IP MTU end to end.

[DarrenB_UK]

Thanks Francesco - ASR1000 series would have been the preferred choice but were cost prohibitive.

I modelled the environment using CSR1000v routers and the MPLS MTU tracks the interface MTU as expected.  If this is set at 1500 bytes, packet loss is observed (sending a 1500 byte packet df-bit set) due to the label overhead shrinking the payload capacity accordingly.

As soon as the HW MTU is set above 1508, no packet loss.  It's just a bit confusing why the "show" commands on the 9300 platform don't exactly reveal the truth about MPLS MTU!

Cheers,

Darren