MTU restriction in L2 Cloud affecting ISIS

Maria Schiaffino · ‎03-14-2018

Hi. We need to configure 6 devices in a network /28 to see each other in ISIS.
4 of them have IOS and 2 of them have IOS XR. The interfaces are subinterfaces of a bundle.
And there is a restriction from the Vendor giving us Layer 2 connectivity that MTU size must be 9000, no higher.
However in the Bundle interface of all the 6 devices we have the MTU set to 9216.
There are other subintefaces not in ISIS on that Interface, that are configured to form BGP peering to get GRE tunnels formed above them.

At this moment this is what we have configured:

XR-ROUTER
interface Bundle-Ether10
bfd address-family ipv4 fast-detect
mtu 9216
interface Bundle-Ether10.2085
ipv4 address 10.10.10.1 255.255.255.240
encapsulation dot1q 2085
router isis CGs
is-type level-2-only
net 49.0001.2170.1001.6020.00
nsf cisco
log adjacency changes
address-family ipv4 unicast
metric-style wide
redistribute connected route-policy CONN2ISIS
!
interface Bundle-Ether10.2085
circuit-type level-2-only
address-family ipv4 unicast
metric 10000000
!

IOS-ROUTER
!
interface Port-channel10
mtu 9216
no ip address
load-interval 30
end
interface Port-channel10.2085
encapsulation dot1Q 2085
ip address 10.10.10.2 255.255.255.240
no ip redirects
no ip proxy-arp
ip flow ingress
ip router isis
isis metric 10000000
end
router isis
net 49.0001.2170.1001.6031.00
is-type level-2-only
area-password NT$ucks
metric-style wide
no hello padding
redistribute connected
redistribute static ip
passive-interface Loopback0

With that configuration the neighbors are stuck in INIT. And we saw that they are sending larger packets (9213) from one end that are not received from the other.
CET: ISIS-Adj: Sending L2 LAN IIH on Port-channel10.2085, length 9213

If we added the command "clns mtu 9000" in the IOS subinterface in two IOS ends, the ISIS neighbor came up.
interface Port-channel10.2085
clns mtu 9000

But this command is not available in IOS XR. There is a command "lsp-mtu" at router ISIS level
RP/0/RSP0/CPU0:cg1.prv2(config-isis)#lsp-mtu ?
<128-4352> Max LSP size in bytes

At the time of testing the IOS XR interface was down, so we are not sure if the neighboring would have came up with IOS interface with clns mtu set. Do you know if it should work?

1) Can you help us to determine if these "clns mtu" in the IOS subinterfaces would make the ISIS neigh come up in the other IOS neigh as well as in XR neigh and that won´t affect any other ISIS neigh on other networks or configurations of other subinterfaces?

2) Also, we think that a proper fix would be to change the Interface MTU to 9000 but that will affect the BGP sessions for the GRE tunnels.
Do you know if changing the MTU only on our side (we don´t have managment of the other end of those tunnels) can make the tunnels go down and never come up?
BTW ip tcp path-mtu-discovery is enabled.

interface Port-channel10
mtu 9216
no ip address
load-interval 30
end
interface Port-channel10.2500
encapsulation dot1Q 2500
ip vrf forwarding VRF10155
ip address 10.254.6.241 255.255.255.254
no ip redirects
no ip proxy-arp
ip flow ingress

interface Tunnel2500
ip vrf forwarding VRF10155
ip address 172.29.0.1 255.255.255.252
ip mtu 1450
ip tcp adjust-mss 1300
ip policy route-map zscaler-tunnels
load-interval 30
keepalive 10 3
tunnel source 10.254.6.241
tunnel destination 10.136.41.41
tunnel vrf VRF10155

SH ip bgp vpnv4 vrf VRF10155 neigh 10.254.6.240 | b Datagrams
Datagrams (max data segment is 9130 bytes):
Rcvd: 5780031 (out of order: 0), with data: 3078735, total data bytes: 58497713
Sent: 5722167 (retransmit: 17), with data: 2761066, total data bytes: 52460280
(c1r1.kis1.se) sescqrt01nt01fm#

(c1r1.kis1.se) sescqrt01nt01fm#ping vrf VRF10155 10.254.6.240 size 9000

Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to 10.254.6.240, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/24/24 ms
(c1r1.kis1.se) sescqrt01nt01fm#ping vrf VRF10155 10.254.6.240 size 9130

Type escape sequence to abort.
Sending 5, 9130-byte ICMP Echos to 10.254.6.240, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
(c1r1.kis1.se) sescqrt01nt01fm#

Thanks in advance for your help!

Maria Schiaffino · ‎04-10-2018

We have changed the MTU size of the Portchannel interfaces in all 6 devices, but ISIS neighbours only came up between IOS neigh and between IOS XR neighbors, but not IOS and IOS XR.

Do you have any suggestion on what to test? What might be wrong or missing either in IOS or XR?
Please find the devices output.

IOS XR

RP/0/RSP0/CPU0:cg1.prv2#sh run int BE10.2085
Tue Apr 10 14:56:11.888 CEST
interface Bundle-Ether10.2085
description One in DK
mtu 9000
ipv4 address 85.x.x.x 255.255.255.240
encapsulation dot1q 2085
!

RP/0/RSP0/CPU0:cg1.prv2#sh run int Bundle-Ether10
Tue Apr 10 14:56:21.076 CEST
interface Bundle-Ether10
description
bfd address-family ipv4 fast-detect
mtu 9000
load-interval 30
!

RP/0/RSP0/CPU0:cg1.prv2#sh isis neigh
Tue Apr 10 14:56:03.800 CEST

IS-IS CGs neighbors:
System Id      Interface        SNPA           State Holdtime Type IETF-NSF
dkccqrt09nt00fm Te0/2/1/3        001b.0def.9380 Up    21       L2   Capable
cg2.bal6       BE10.2085        70e4.224d.0b93 Up    22       L2   Capable

Total neighbor count: 2

RP/0/RSP0/CPU0:cg1.prv2#sh isis int Bundle-Ether10.2085
Tue Apr 10 14:56:42.227 CEST

Bundle-Ether10.2085         Enabled
Adjacency Formation:      Enabled
Prefix Advertisement:     Enabled
IPv4 BFD:                 Disabled
IPv6 BFD:                 Disabled
BFD Min Interval:         150
BFD Multiplier:           3

Circuit Type:             level-2-only
Media Type:               LAN
Circuit Number:           5

Level-2
    Adjacency Count:        1
    LAN ID:                 cg1.prv2.05
    Priority (Local/DIS):   64/64
    Next LAN IIH in:        2 s
    LSP Pacing Interval:    33 ms
    PSNP Entry Queue Size: 0

CLNS I/O
    Protocol State:         Up
    MTU:                    8979
    SNPA:                   78ba.f92d.11bb
    Layer-2 MCast Groups Membership:
      All Level-2 ISs:      Yes

IPv4 Unicast Topology:    Enabled
    Adjacency Formation:    Running
    Prefix Advertisement:   Running
    Metric (L1/L2):         0/10000000
    Weight (L1/L2):         0/0
    MPLS Max Label Stack:   3
    MPLS LDP Sync (L1/L2): Disabled/Disabled

IPv4 Address Family:      Enabled
    Protocol State:         Up
    Forwarding Address(es): 85.89.227.81
    Global Prefix(es):      85.89.227.80/28

LSP transmit timer expires in 0 ms
LSP transmission is idle
Can send up to 9 back-to-back LSPs in the next 0 ms

RP/0/RSP0/CPU0:cg1.prv2#SH IM DAtabase int Bundle-Ether10.2085
Tue Apr 10 15:20:17.235 CEST

View: OWN - Owner, L3P - Local 3rd Party, G3P - Global 3rd Party, LDP - Local Data Plane
      GDP - Global Data Plane, RED - Redundancy, UL - UL

Node 0/RSP0/CPU0 (0x41)

Interface Bundle-Ether10.2085, ifh 0x08000160 (up, 9000)
Interface flags:          0x00000000000205d7 (REPLICATED|IFINDEX
                            |SUP_NAMED_SUB|BROADCAST|VIRTUAL|CONFIG|VIS|DATA
                            |CONTROL)
Encapsulation:            dot1q
Interface type:           IFT_VLAN_SUBIF
Control parent:           Bundle-Ether10
Data parent:              Bundle-Ether10
Views:                    UL|GDP|G3P|L3P|OWN

Protocol        Caps (state, mtu)
--------        -----------------
None            vlan_jump (up, 9000)
None            dot1q (up, 9000)
arp             arp (up, 8982)
clns            clns (up, 8982)
ipv4            ipv4 (up, 8982)

***

IOS

(c1r1.kis1.se) sescqrt01nt01fm#sh run int Port-channel10
Building configuration...

Current configuration : 119 bytes
!
interface Port-channel10
description In SE
mtu 9000
no ip address
load-interval 30
end

(c1r1.kis1.se) sescqrt01nt01fm#sh run int Po10.2085
Building configuration...

Current configuration : 252 bytes
!
interface Port-channel10.2085
description SE
encapsulation dot1Q 2085
ip address 85.x.x.x 255.255.255.240
no ip redirects
no ip proxy-arp
ip flow ingress
ip router isis
isis metric 10000000
end
(c1r1.kis1.se) sescqrt01nt01fm#sh isis neig

System Id      Type Interface   IP Address      State Holdtime Circuit Id
cg1.prv2       L2   Po10.2085   85.89.227.81    INIT 9        cg1.prv2.05
cg2.bal6       L2   Po10.2085   85.89.227.82    INIT 20       cg2.bal6.05
cp5.kis1       L2   Gi6/1       85.89.240.30    UP    29       sescqrt01nt01f.01
cp6.sol1       L2   Gi6/2       85.89.240.38    UP    28       sescqrt01nt01f.02
fiheqrt01nt00fmL2   Po10.2085   85.89.227.85    UP    23       fiheqrt02nt00f.04
fiheqrt02nt00fmL2   Po10.2085   85.89.227.86    UP    7        fiheqrt02nt00f.04
sescqrt02nt01fmL2   Po10.2085   85.89.227.84    UP    29       fiheqrt02nt00f.04

(c1r1.kis1.se) sescqrt01nt01fm#sh clns int Port-channel10.2085
Port-channel10.2085 is up, line protocol is up
Checksums enabled, MTU 8997, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 9 seconds
Routing Protocol: IS-IS
    Circuit Type: level-1-2
    Interface number 0x2, local circuit ID 0x3
    Level-2 Metric: 10000000, Priority: 64, Circuit ID: fiheqrt02nt00f.04
    DR ID: fiheqrt02nt00f.04
    Level-2 IPv6 Metric: 10
    Number of active level-2 adjacencies: 3
    Next IS-IS LAN Level-2 Hello in 2 seconds
    No hello padding

bersoare · ‎04-10-2018

It looks like you still have a MTU mismatch.

Could you please collect show ipv4 int from the XR, and a show ip int on the IOS device?

The MTU values should match, but the XR takes into account the 14 bytes IOS won't consider. So, if you have a MTU of 1500 in the IOS box, you should have 1514 in the XR, for example.

Maria Schiaffino · ‎04-11-2018

Hi,

Thanks for your reply. Here is the output you are asking for:

RP/0/RSP0/CPU0:cg1.prv2#sh ipv4 int bundle-ether 10.2085

Wed Apr 11 13:53:19.257 CEST

Bundle-Ether10.2085 is Up, ipv4 protocol is Up

Vrf is default (vrfid 0x60000000)

Internet address is 85.89.227.81/28

MTU is 9000 (8982 is available to IP)

Helper address is not set

Multicast reserved groups joined: 224.0.0.2 224.0.0.1

Directed broadcast forwarding is disabled

Outgoing access list is not set

Inbound common access list is not set, access list is not set

Proxy ARP is disabled

ICMP redirects are never sent

ICMP unreachables are always sent

ICMP mask replies are never sent

Table Id is 0xe0000000

RP/0/RSP0/CPU0:cg1.prv2#

One thing that I noticed now is that we are receiving the following error message since the mtu changed to 9000 in XR:

isis[1010]: %ROUTING-ISIS-6-ERR_DUPID : Duplicate System ID 2170.1001.6020 already used by Local System detected in IIH received on Bundle-Ether10.2085 SNPA 78ba.f92d.11bb

And I only see it once on a different day on the other XR. And both have different System IDs. What is the System ID that seems to be duplicated? Because I cannot find any other device with 2170.1001.6020? Is this related to the fact of ISIS adjency not coming up between IOS and XR?

Thanks again!

bersoare · ‎04-11-2018

Hi Maria,

Now we can clearly see the MTU setting differs:

XR:

MTU is 9000 (8982 is available to IP)

clns clns (up, 8982) <<

IOS:

sescqrt01nt01fm#sh clns int Port-channel10.2085
Port-channel10.2085 is up, line protocol is up
Checksums enabled, MTU 8997 <<

How is the actual MTU configured in the IOS device? check it with a show ip int Port-channel10.2085, then tune it up until you see the same value on both ends.

Regarding this message, i have only seen those in loop scenarios (where a multicast sent by a router would come back in a different interface), but this is not related to adj establishment.

This message says that a system id of 2170.1001.6020 (which is our local ID) was seen in the hello packets, coming from the node with SNPA (typically the mac-address) of 78ba.f92d.11bb

Check the neighbor's SNPA with the command show clns neighbor.

Maria Schiaffino · ‎04-11-2018

Hi,

The MTU in the IOS device subinterface is taken from the Interface:

IOS_device#show ip int Port-channel10.2085

Port-channel10.2085 is up, line protocol is up

Internet address is 85.x.x.x/28

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 9000 bytes

So, as I see it, I have two options to fix this. Correct me if I am wrong:

1)

In XR do

interface Bundle-Ether10.2085

no mtu 9000

interface Bundle-Ether10

mtu 9018

but this will cause all the subinterfaces to be reconfigured (the intefaces that are part of the port channel will go down and back again) then the BGP peers in the other subinterfaces will flap.

2)

In IOS do:

interface Port-channel10.2085

clns mtu 8983

And in XR

interface Bundle-Ether10.2085

no mtu 9000

This would only fix the ISIS adjencency between XR and IOS and also IOS - IOS adj should remain working. This will still have mtu 9000 both in XR and IOS interface level. I understand this is not the best option but would cause the less impact and fix ISIS, right?

Regarding the DUP System ID message, it is strange because 2170.1001.6020 as you said is our local ID and the SNPA (typically the mac-address) of 78ba.f92d.11bb is the one of the Local subinteface:

RP/0/RSP0/CPU0:cg1.prv2#sh int bundle-ether 10

Wed Apr 11 15:24:30.928 CEST

Bundle-Ether10 is up, line protocol is up

Interface state transitions: 15

Hardware is Aggregated Ethernet interface(s), address is 78ba.f92d.11bb

...

RP/0/RSP0/CPU0:cg1.prv2#sh int bundle-ether 10.2085

Wed Apr 11 15:38:19.648 CEST

Bundle-Ether10.2085 is up, line protocol is up

Interface state transitions: 3

Hardware is VLAN sub-interface(s), address is 78ba.f92d.11bb

...

RP/0/RSP0/CPU0:cg1.prv2#sh run router isis

Wed Apr 11 15:35:26.722 CEST

router isis CGs

is-type level-2-only

net 49.0001.2170.1001.6020.00

>From an IOS device I see:

(c1r2.sol1.se) sescqrt02nt01fm#show clns neighbor

System Id Interface SNPA State Holdtime Type Protocol

cg1.prv2 Po10.2085 78ba.f92d.11bb Init 7 L2 IS-IS

cg2.bal6 Po10.2085 70e4.224d.0b93 Init 23 L2 IS-IS

cp5.kis1 Gi6/2 0014.1b96.9fc0 Up 27 L2 IS-IS

cp6.sol1 Gi6/1 0015.2c50.2000 Up 25 L2 IS-IS

fiheqrt01nt00fmPo10.2085 1cdf.0fc9.afc0 Up 26 L2 IS-IS

fiheqrt02nt00fmPo10.2085 f872.eab8.4780 Up 8 L2 IS-IS

sescqrt01nt01fmPo10.2085 001b.8f12.57c0 Up 20 L2 IS-IS

(c1r2.sol1.se) sescqrt02nt01fm#

I don´t think this is because we use a /28 for the subnet including the 6 devices, right?

Thanks again for your time!

bersoare · ‎04-11-2018

Okay, so stepping back a little, you said that you need to have at most 9000 MTU in this L2, right?

Being so, you can go with the following:

XR

MTU 9014 - this would give us 9000 for both CLNS and IPv4

IOS

MTU 9003 - This, by default, would give an IP mtu of 9003 and 9000 for CLNS. You can also set IP mtu for 9000 and manually set the clns MTU if you want.

To understand better the log messages, a debug or packet capture would be the best way to move on.

Maria Schiaffino · ‎04-11-2018

Ok, I will need a window to do the tests and debugs. There is another link with issues which is also running ISIS so it might be disturbing the troubleshooting. I will post the results when I can do the tests.

Thank you very much for your help and time!

bersoare · ‎04-17-2018

Hello Maria,

You are very welcome. Please let me know the results.

Rodrigo Octaviano · ‎05-07-2018

Hello Maria, how are you doing?

Do you can do this test? Because I'm with the same problem here....

Thks!

Maria Schiaffino · ‎05-07-2018

Hi there,

just today I was able to do the tests. The Duplicated ID message disappeared as soon the provider shut down one of the interfaces going to their devices which was part of our bundle. We have two interfaces in a bundle in the same devices (XR) going to two different Juniper devices at the providers end. So, this mean there was a loop at the provider ends.

And today, (with that link down still), I added "clns mtu 8982" to the subinterface po10.2085 in the IOS devices and the ISIS neighbors came up. So, we have "mtu 9000" in XR Bundle interface and "clsn mtu 8982" in the IOS subinterface. And ISIS is up!!

Hope this works for you as well!