Maximum MTU over a pseudowire between two IOS XR NCS5500

[j.restaino]

Dear All,

I would like to help me to understand why we are having the following limitation in the MTU. I have two routers interconnected to each other using MPLS, ISIS an MTU of 9300, between them. I configured a pseudowire using an MTU of 9216 in the service sub-interface, and 9300 at the MTU of the interface. When pinging from an interface connected to the service with MTU 9646, we have that the maximum MTU is of 9264.

Configuration

CE1 Router

interface TenGigE0/1/0/24/3
 mtu 9646
!
interface TenGigE0/1/0/24/3.201
 mtu 9646
 ipv4 address 10.0.0.1 255.255.255.252
 encapsulation dot1q 201
!

CE2

interface TenGigE0/1/0/24/3
 mtu 9646
!
interface TenGigE0/1/0/24/3.201
 mtu 9646
 ipv4 address 10.0.0.2 255.255.255.252
 encapsulation dot1q 201

PE1 Router

interface TenGigE0/1/0/24/2
 mtu 9300
 load-interval 30
!
interface TenGigE0/1/0/24/2.100009991 l2transport
 encapsulation dot1q 201
 rewrite ingress tag pop 1 symmetric
 mtu 9216
!
l2vpn
 pw-class CLASS-L2L
 encapsulation mpls
  protocol ldp
  transport-mode ethernet
 !
!
 xconnect group PW-L2L
  p2p EVC-10000999
  interface TenGigE0/1/0/24/2.100009992
  neighbor ipv4 192.168.197.13 pw-id 10000999
   pw-class CLASS-L2L
 !
!

PE2 Router

interface TenGigE0/1/0/24/2
 mtu 9300
 load-interval 30
!
interface TenGigE0/1/0/24/2.100009992 l2transport
 encapsulation dot1q 201
 rewrite ingress tag pop 1 symmetric
 mtu 9216
 service-policy input Policy_10000999
!
l2vpn
 pw-class CLASS-L2L
  encapsulation mpls
   protocol ldp
   transport-mode ethernet
 !
!
 xconnect group PW-L2L
  p2p EVC-10000999
   interface TenGigE0/1/0/24/2.100009991
   neighbor ipv4 192.168.197.14 pw-id 10000999
    pw-class CLASS-L2L
   !

PING Test

RP/0/RP0/CPU0:brpan4.mvd#ping 10.0.0.1 df-bit size 9265
Mon Jul 3 15:22:21.790 GMT+3
Type escape sequence to abort.
Sending 5, 9265-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:brpan4.mvd#ping 10.0.0.1 df-bit size 9264
Mon Jul 3 15:22:59.053 GMT+3
Type escape sequence to abort.
Sending 5, 9264-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms

PW MTU 9198

RP/0/RP0/CPU0:PE2#show l2vpn xconnect pw-id 10000999 detail
Group PW-L2L_INTERNACIONALES, XC EVC-10000999, state is up; Interworking none
Description: "EVC-10000999 - test"
AC: TenGigE0/1/0/24/2.100009992, state is up
Type VLAN; Num Ranges: 1
Rewrite Tags: []
VLAN ranges: [201, 201]
MTU 9198; XC ID 0x800001; interworking none
Statistics:
packets: received 11843, sent 11795
bytes: received 14051188, sent 13692483
drops: illegal VLAN 0, illegal length 0
PW: neighbor 192.168.197.13, PW ID 10000999, state is up ( established )
PW class CLASS-L2L, XC ID 0xc000000c
Encapsulation MPLS, protocol LDP
Source address 192.168.197.14
PW type Ethernet, control word disabled, interworking none
PW backup disable delay 0 sec
Sequencing not set
Ignore MTU mismatch: Disabled
Transmit MTU zero: Disabled
LSP : Up

PW Status TLV in use
MPLS Local Remote
------------ ------------------------------ -----------------------------
Label 24121 24123
Group ID 0x2002298 0x2002298
Interface TenGigE0/1/0/24/2.100009992 TenGigE0/1/0/24/2.100009991
MTU 9198 9198
Control word disabled disabled
PW type Ethernet Ethernet
VCCV CV type 0x2 0x2
(LSP ping verification) (LSP ping verification)
VCCV CC type 0x6 0x6
(router alert label) (router alert label)
(TTL expiry) (TTL expiry)
------------ ------------------------------ -----------------------------
Incoming Status (PW Status TLV):
Status code: 0x0 (Up) in Notification message
Outgoing Status (PW Status TLV):
Status code: 0x0 (Up) in Notification message
MIB cpwVcIndex: 3221225484
Create time: 29/06/2023 16:40:29 (3d23h ago)
Last time status changed: 30/06/2023 17:20:41 (2d22h ago)
Last time PW went down: 30/06/2023 17:19:14 (2d22h ago)
Statistics:
packets: received 11795, sent 11843
bytes: received 13692483, sent 14051188

[M02@rt37 · Accepted solution]

Hello @j.restaino,

Given the configuration and output you provided, it appears that the effective MTU for the pseudowire (PW) is 9216, as indicated by the output: "MTU 9216" in the PW configuration on both PE1 and PE2 routers.

Therefore, the effective MTU for the entire path, considering the lowest MTU value along the path, is 9216. This is lower than the MTU of the CE1 and CE2 interfaces (9646) but higher than the MTU of the PE1 and PE2 interfaces (9300).

The limitation you're experiencing with a maximum ping packet size of 9264 is likely due to additional protocol overhead or encapsulation added by the network devices. The effective MTU of 9216 should allow for the successful transmission of packets without fragmentation, as long as the packet size does not exceed 9216 bytes, considering any additional overhead.

Given the ping test results you provided, it shows that the maximum MTU that can be successfully pinged is 9264, which is higher than the expected effective MTU of 9216. This discrepancy could be due to additional protocol overhead or encapsulation added by the network devices.

It appears that the effective MTU for the pseudowire is indeed 9216, and the maximum successful ping packet size of 9264 could be attributed to additional overhead.

[ammahend · Accepted solution]

Is this causing any application related issue, the ping failure seem obvious since you are sending packets higher than MTU on interface with DF bit set, so the interface can not fragment and has to drop

on PE1 or PE2 where MTU is 9300 if you do a capture find a tcp syn, expand tcp header>tcp option>Maximum segment size (MSS), what is this value ?

if its well within this limit where with all additional headed+MSS you do not exceed lowest MTU in path then, I don't think you will experience any issue or fragmentation.

Default value for Jumbo frames can be different for different vendors or even products.

[j.restaino · Accepted solution]

Thanks for your answers @ammahend and M02@rt37  i think that now i have the answer.

All the routers involved are Cisco NCS5500 with IOS XR 7.3.1.

I think that the PW MTU is 9198 because the nogotiation is made with the sub interface MTU considering Ethernet and MPLS headers (9216 - 14 - 4). But the data tráffic MTU i think that is limited by the interface 9300, with this value MTU we need to disccount the next headers (Ethernet Data, Ehternet between PE, MPLS inner header and MPLS outer header) so the maximun MTU is 9300 - 14 - 14 - 4 - 4 = 9264. I need to add outer MPLS header because i am using Explicit Null Tag.