Routing problem with MPLS/ISIS

[CliveG]

Hi,

We have a 3 x DC architecture that can be summarised as per below:

DC1 - West London
DC2 - Telehouse West
DC3 - Telehouse North
DC1 has an ethernet metro line directly to DC2 and DC3 and likewise DC2 and DC3 have a X-Connect within Telehouse, thus forming a tringle.

Originally all of our external traffic from West London (DC1) exited via Ethernet metro to Telehouse North.
To upgrade the connectivity and create DC redundancy, I organised better bandwidth uplinks from Telehouse West.
At DC2 and DC3 as well as transit connectivity we also have LINX Connectivity.
I now have traffic exiting our network via Telehouse West, thus allowing us to upgrade Telehouse North, or should be, except for one issue as explained below:

Traffic destined for the outside world is still traversing Telehouse North (DC3) first and then Telehouse West (DC2) instead of direct to Telehouse West.
Unfortunatly, I am on a network that cannot be "Down" for any reason (other than the obvious complete hardware failure etc).

The internal routing is being handled by ISIS/MPLS and given that I am not a Cisco expert (worked on Juniper mainly over the past several years), I was hoping someone may be able to aid in resolving how I can get the routing changed to go direct over the ethernet metro from DC1 to DC2 without going to DC3 first.

On top of that, I cannot make use of the upgraded LINX connectivity at DC2 because of the above issue. The BGP open packet wants to exit via DC3 and LINX Collector naturally sees this from the wrong address and drops the packets.

Here is the ISIS/MPLS configs I can see on DC1:

router isis 40000
vrf VFM
net 49.4000.4013.4013.00
metric-style wide
redistribute connected

router isis

router isis <ASN>
net 49.3323.9613.9613.00
metric-style wide
redistribute connected
redistribute static ip
maximum-paths 8
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

And DC3:

router isis <ASN>
net 49.3323.9612.9614.00
metric-style wide
metric 100
redistribute connected
redistribute static ip
maximum-paths 8
distance 100 clns
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

And DC2:

router isis 40000
vrf VFM
net 49.4000.4011.4011.00
metric-style wide
redistribute connected

router isis MULTICAST
net 31.3131.3131.3131.00
metric-style wide

router isis <ASN>
net 49.3323.9611.9611.00
metric-style wide
metric 100
redistribute connected
redistribute static ip
maximum-paths 8
distance 100 clns
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

If any other information is required then please let me know and I will supply.

Many thanks

[CliveG]

Hi Harold,

My apologies. I neglected to mention that all three DCs are running the following:

DC1 - 2 x C6880-X in VSS - Port-Channel to DC2 and direct ethernet metro link to DC3
DC2 - 2 x C6880-X in VSS - Port-Channel to DC1 Ethernet Metro and X-Connect to DC3
DC3 - 2 x C6880-X in VSS - Direct connect to DC1 Ethernet Metro and X-Connect to DC2

I will get the output of the commands you have asked for in the morning.

Again, thank you

[CliveG]

Hi Harold,

Please find shown below the output of the commands you requested:

DC2:
show switch virtual:
Switch mode : Virtual Switch
Virtual switch domain number : 10
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby

Show switch virtual role:
RRP information for Instance 1

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Priority Role Local Remote
Number Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 1 UP 110(110) ACTIVE 0 0
REMOTE 2 UP 100(100) STANDBY 4620 3355

Peer 0 represents the local switch

Flags : V - Valid


In dual-active recovery mode: No

DC3:
show switch virtual:
Switch mode : Virtual Switch
Virtual switch domain number : 30
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby

show switch virtual role:
RRP information for Instance 1

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Priority Role Local Remote
Number Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 1 UP 100(100) ACTIVE 0 0
REMOTE 2 UP 100(100) STANDBY 895 3436

Peer 0 represents the local switch

Flags : V - Valid

In dual-active recovery mode: No

If anything else is required then please let me know.

[Harold Ritter]

Hi @CliveG ,

That confirms that DC2 and DC3 are running as a single VSS. This explains why DC1 is only showing an ISIS neighbor relationship to DC3 (active virtual switch).

I would suggest you reconsider whether VSS is the most appropriate way to achieve redundancy for your network. 

Regards,

[CliveG]

Hi Harold,
I do not think this is the case.

In DC2 we have 2 x C6880-X that are linked together as a VSS and in DC3 we have 2 x C6880-X tied together as a VSS.

There is no configuration between the two DCs to state any VSS is present between them. The domain numbers for the VSS are completely different.

Also, the LAB shows exactly the same VSS information and yet functions correctly.

I have attached a diagram to show exactly what the topology is.

May be there is a difference in the connectivity that is affecting something (as in the LAG as opposed to direct).

As can be seen from the diagram, I am trying to get the 2 x 10gbps Links to LINX up with eBGP but I cannot because of all these routing issues. DC2 is the better upstream connectivity currently. I want to get this routing correct so I can upgrade DC3 to the same.

Many thanks

[Harold Ritter]

Hi @CliveG ,

Thanks for the additional information. Glad to know you are not running VSS between the DC2 and DC3.

We need to find out why the ISIS neighbor does not come up.

Can you provide the output of "show runn interface <lag interface name>" and "show clns proto" from both DC1 and DC3?

Regards,

[CliveG]

Hi Harold,

There is not a LAG (as the diagram shows) between DC1 and DC3. I want to upgrade this to a LAG but I cannot because all the routing goes that way. Once I can resolve this I can get the LAG connected and configured between DC1 and DC3 and I can also get the eBGP open packet to get to LINX from DC2 instead of it going via DC3 and getting dropped.

Here is the requested information:
DC1:
C6880-VSS-DC1#show run interface te2/5/10
interface TenGigabitEthernet2/5/10
description COLT line to THN
no switchport
mtu 8000
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::8:13/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>

C6880-VSS-DC1#show clns protocol

IS-IS Router: 40000
System Id: 4000.4013.4013.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

IS-IS Router: <Null Tag>
System Id: 0000.0000.0000.00 IS-Type: level-1-2
Manual area address(es):
Routing for area address(es):
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: level-1-2
Accept narrow metrics: level-1-2
Generate wide metrics: none
Accept wide metrics: none

IS-IS Router: <ASN>
System Id: 3323.9613.9613.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
Interfaces supported by IS-IS:
TenGigabitEthernet2/5/10 - OSI - IP - IPv6
Port-channel512 - OSI - IP - IPv6
Port-channel510 - OSI - IP - IPv6
Loopback2 - OSI - IP
Loopback1 - OSI - IP - IPv6
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

DC3:
C6880X-VSS-DC3#show run interface te1/5/16
Building configuration...

Current configuration : 277 bytes
!
interface TenGigabitEthernet1/5/16
description COLT line to CHI
no switchport
mtu 8000
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::8:14/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>

C6880X-VSS-DC3#show clns protocol

IS-IS Router: <ASN>
System Id: 3323.9612.9614.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
Interfaces supported by IS-IS:
Vlan2017 - IP
Vlan1105 - OSI - IP - IPv6
TenGigabitEthernet1/5/16 - OSI - IP - IPv6
Port-channel211 - OSI - IP - IPv6
Port-channel210 - OSI - IP - IPv6
Loopback0 - OSI - IP - IPv6
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 100
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

Thank you

[Harold Ritter]

Sorry I meant "Can you provide the output of "show runn interface <lag interface name>" and "show clns proto" from both DC1 and DC2?"

Regards,

[CliveG]

Hi Harold,

Please find below the readout for DC2 and the LAG config for DC1 to DC2 connection on DC1:

DC2:
C6880-VSS-DC2#show run interface port-channel 512
Building configuration...

Current configuration : 292 bytes
!
interface Port-channel512
description Colt to DC1_VSS_TE 1/5/15
no switchport
mtu 9216
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::4:11/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>
end

C6880-VSS-DC2#show clns protocol

IS-IS Router: 40000
System Id: 4000.4011.4011.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

IS-IS Router: MULTICAST
System Id: 3131.3131.3131.00 IS-Type: level-1-2
Manual area address(es):
31
Routing for area address(es):
31
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

IS-IS Router: <ASN>
System Id: 3323.9611.9611.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
Interfaces supported by IS-IS:
TenGigabitEthernet1/5/10 - IP
Port-channel512 - OSI - IP - IPv6
Port-channel511 - OSI - IP - IPv6
Loopback0 - IP
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 100
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

You will also need the show run for the LAG interface to DC2 from DC1:

DC1:
C6880-VSS-DC1#show run interface port-channel 512
Building configuration...

Current configuration : 268 bytes
!
interface Port-channel512
description To-DC2-LAG
no switchport
mtu 9216
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::4:13/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>
end

Many thanks

[Harold Ritter]

Everything looks good so far.

Can you also provide a "sh clns int Port-channel512" from DC1 and DC2.

Also, I see the MTU you set on the port channel interface is 9216. Can you make sure you can ping from DC1 to DC2 to port channel interface ip address using that MTU size ("ping x.x.x.x size 9216 df-bit"). If not, this could be the reason why the ISIS neighbour does not come up.

Regards,

[CliveG]

Hi Harold,

Many thanks for all your help so far.

Here is the requested information:

DC1:
C6880-VSS-DC1#show clns int port-channel 512
Port-channel512 is up, line protocol is up
Checksums enabled, MTU 9213, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
RDPDUs enabled, min. interval 100 msec., Addr Mask enabled
Congestion Experienced bit set at 4 packets
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 32 seconds
Routing Protocol: IS-IS (ASN)
Circuit Type: level-1-2
Interface number 0x3, local circuit ID 0x2
Level-1 Metric: 10, Priority: 64, Circuit ID: C6880-VSS-DC1.02
DR ID: C6880-VSS-CHI.02
Level-1 IPv6 Metric: 10
Number of active level-1 adjacencies: 0
Level-2 Metric: 10, Priority: 64, Circuit ID: C6880-VSS-DC1.02
DR ID: 0000.0000.0000.00
Level-2 IPv6 Metric: 10
Number of active level-2 adjacencies: 0
Next IS-IS LAN Level-1 Hello in 1 seconds
Next IS-IS LAN Level-2 Hello in 3 seconds

DC2:
C6880-VSS-DC2#show clns int port-channel 512
Port-channel512 is up, line protocol is up
Checksums enabled, MTU 9213, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
RDPDUs enabled, min. interval 100 msec., Addr Mask enabled
Congestion Experienced bit set at 4 packets
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 20 seconds
Routing Protocol: IS-IS (ASN)
Circuit Type: level-1-2
Interface number 0x2, local circuit ID 0x2
Level-1 Metric: 100, Priority: 64, Circuit ID: C6880-VSS-DC2.02
DR ID: C6880-VSS-THE.02
Level-1 IPv6 Metric: 10
Number of active level-1 adjacencies: 0
Level-2 Metric: 100, Priority: 64, Circuit ID: C6880-VSS-DC2.02
DR ID: 0000.0000.0000.00
Level-2 IPv6 Metric: 10
Number of active level-2 adjacencies: 0
Next IS-IS LAN Level-1 Hello in 377 milliseconds
Next IS-IS LAN Level-2 Hello in 8 seconds

Don't suppose the metrics have anything to do with this? One side is 100 and the other is 10 with no adjacencies showing?

Many thanks Harold. Very much appreciated.

[Harold Ritter]

Hi @CliveG ,

> Don't suppose the metrics have anything to do with this? One side is 100 and the other is 10 with no adjacencies showing?

This is not the reason the adjacency is not coming up but you might want to change that, unless there is a specific reason to use different metrics on either side of the port channel.

Also, I see the MTU you set on the port channel interface is 9216. Can you make sure you can ping from DC1 to DC2 to port channel interface ip address using that MTU size ("ping x.x.x.x size 9216 df-bit"). If not, this could be the reason why the ISIS neighbour does not come up.

Regards,

[CliveG]

Hi Harold,

The maximum packet size I can ping with the df-bit set is MTU 9168.

I understand what you mean by the lack of possible connectivity, but here is what makes no sense then. In the LAB I have the following configured on the LAG between DC1 and DC2:

mtu 9216 at both ends and the is-is neighbor is UP as shown below (lab):

Lab-DC1 "show isis neighbors" output:
DC2-VSS-01 L1 Po512 xxx.xxx.xxx.xxx UP 8 LAB-DC2-VSS-01.02
DC2-VSS-01 L2 Po512 xxx.xxx.xxx.xxx UP 7 LAB-DC2-VSS-01.02

And here is the result of the ping test:
LAB-DC1-VSS-01#ping <DC2 LAG Interface address> size 9216 df-bit
Type escape sequence to abort.
Sending 5, 9216-byte ICMP Echos to xxx.xxx.xxx.xxx, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

Something very strange is happening and I cannot work out what it is. I cannot see what would cause the ping to work in the lab but not on the live network. They are both C6880-X switches in a VSS, they both have the same IOS, they both have the same configs and the ports are all cables up the same (within the same 2 racks - this is the only difference - the ethernet metro etc).

Is there anything else I could look at please?

Thanks

[Harold Ritter · Accepted solution]

Hi @CliveG ,

> The maximum packet size I can ping with the df-bit set is MTU 9168.

It looks like we found the issue. This certainly explain why the hello packets do reach between DC1 and DC2.

> mtu 9216 at both ends and the is-is neighbor is UP as shown below (lab):

Bear in mind that in the lab you control the underlay transport (probably back to back between DC1 and DC2). Who provides the transport between the production DC1 and DC2? Most probably a 3rd party and they certainly have specifications on the maximum MTU size they support in their transport network.

As a test, you might want to configure "no isis hello padding always" on the port channel interface on both side and verify that the adjacency comes up. But in the end, the real solution is to adjust the MTU to reflect what is supported by the underlay network.

Regards,

[CliveG]

Hi Harold,

Many thanks for all your help regarding this issue. It is very appreciated.

I have already put a plan in place to lower the MTU to 8000 as per the link to DC3 from DC1. The original MTU was set by the original network engineer and I simpy copied that MTU to the LAG. This would suggest to me that the connectivity between DC1 and DC2 never worked and was never tested.

I will close this case with your solution.

Again, many thanks for your time and effort helping resolve the issue.

[Harold Ritter]

You are very welcome @CliveG. Thanks for the feedback and please let us know if there is anything else we can do to help