Routing problem with MPLS/ISIS

[CliveG]

Hi,

We have a 3 x DC architecture that can be summarised as per below:

DC1 - West London
DC2 - Telehouse West
DC3 - Telehouse North
DC1 has an ethernet metro line directly to DC2 and DC3 and likewise DC2 and DC3 have a X-Connect within Telehouse, thus forming a tringle.

Originally all of our external traffic from West London (DC1) exited via Ethernet metro to Telehouse North.
To upgrade the connectivity and create DC redundancy, I organised better bandwidth uplinks from Telehouse West.
At DC2 and DC3 as well as transit connectivity we also have LINX Connectivity.
I now have traffic exiting our network via Telehouse West, thus allowing us to upgrade Telehouse North, or should be, except for one issue as explained below:

Traffic destined for the outside world is still traversing Telehouse North (DC3) first and then Telehouse West (DC2) instead of direct to Telehouse West.
Unfortunatly, I am on a network that cannot be "Down" for any reason (other than the obvious complete hardware failure etc).

The internal routing is being handled by ISIS/MPLS and given that I am not a Cisco expert (worked on Juniper mainly over the past several years), I was hoping someone may be able to aid in resolving how I can get the routing changed to go direct over the ethernet metro from DC1 to DC2 without going to DC3 first.

On top of that, I cannot make use of the upgraded LINX connectivity at DC2 because of the above issue. The BGP open packet wants to exit via DC3 and LINX Collector naturally sees this from the wrong address and drops the packets.

Here is the ISIS/MPLS configs I can see on DC1:

router isis 40000
vrf VFM
net 49.4000.4013.4013.00
metric-style wide
redistribute connected

router isis

router isis <ASN>
net 49.3323.9613.9613.00
metric-style wide
redistribute connected
redistribute static ip
maximum-paths 8
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

And DC3:

router isis <ASN>
net 49.3323.9612.9614.00
metric-style wide
metric 100
redistribute connected
redistribute static ip
maximum-paths 8
distance 100 clns
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

And DC2:

router isis 40000
vrf VFM
net 49.4000.4011.4011.00
metric-style wide
redistribute connected

router isis MULTICAST
net 31.3131.3131.3131.00
metric-style wide

router isis <ASN>
net 49.3323.9611.9611.00
metric-style wide
metric 100
redistribute connected
redistribute static ip
maximum-paths 8
distance 100 clns
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

If any other information is required then please let me know and I will supply.

Many thanks

[CliveG]

Hi Harold,

My apologies. I neglected to mention that all three DCs are running the following:

DC1 - 2 x C6880-X in VSS - Port-Channel to DC2 and direct ethernet metro link to DC3
DC2 - 2 x C6880-X in VSS - Port-Channel to DC1 Ethernet Metro and X-Connect to DC3
DC3 - 2 x C6880-X in VSS - Direct connect to DC1 Ethernet Metro and X-Connect to DC2

I will get the output of the commands you have asked for in the morning.

Again, thank you

[CliveG]

Hi Harold,

Please find shown below the output of the commands you requested:

DC2:
show switch virtual:
Switch mode : Virtual Switch
Virtual switch domain number : 10
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby

Show switch virtual role:
RRP information for Instance 1

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Priority Role Local Remote
Number Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 1 UP 110(110) ACTIVE 0 0
REMOTE 2 UP 100(100) STANDBY 4620 3355

Peer 0 represents the local switch

Flags : V - Valid


In dual-active recovery mode: No

DC3:
show switch virtual:
Switch mode : Virtual Switch
Virtual switch domain number : 30
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby

show switch virtual role:
RRP information for Instance 1

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Priority Role Local Remote
Number Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 1 UP 100(100) ACTIVE 0 0
REMOTE 2 UP 100(100) STANDBY 895 3436

Peer 0 represents the local switch

Flags : V - Valid

In dual-active recovery mode: No

If anything else is required then please let me know.

[Harold Ritter]

Hi @CliveG ,

That confirms that DC2 and DC3 are running as a single VSS. This explains why DC1 is only showing an ISIS neighbor relationship to DC3 (active virtual switch).

I would suggest you reconsider whether VSS is the most appropriate way to achieve redundancy for your network. 

Regards,

[CliveG]

Hi Harold,
I do not think this is the case.

In DC2 we have 2 x C6880-X that are linked together as a VSS and in DC3 we have 2 x C6880-X tied together as a VSS.

There is no configuration between the two DCs to state any VSS is present between them. The domain numbers for the VSS are completely different.

Also, the LAB shows exactly the same VSS information and yet functions correctly.

I have attached a diagram to show exactly what the topology is.

May be there is a difference in the connectivity that is affecting something (as in the LAG as opposed to direct).

As can be seen from the diagram, I am trying to get the 2 x 10gbps Links to LINX up with eBGP but I cannot because of all these routing issues. DC2 is the better upstream connectivity currently. I want to get this routing correct so I can upgrade DC3 to the same.

Many thanks

[Harold Ritter]

Hi @CliveG ,

Thanks for the additional information. Glad to know you are not running VSS between the DC2 and DC3.

We need to find out why the ISIS neighbor does not come up.

Can you provide the output of "show runn interface <lag interface name>" and "show clns proto" from both DC1 and DC3?

Regards,

[CliveG]

Hi Harold,

There is not a LAG (as the diagram shows) between DC1 and DC3. I want to upgrade this to a LAG but I cannot because all the routing goes that way. Once I can resolve this I can get the LAG connected and configured between DC1 and DC3 and I can also get the eBGP open packet to get to LINX from DC2 instead of it going via DC3 and getting dropped.

Here is the requested information:
DC1:
C6880-VSS-DC1#show run interface te2/5/10
interface TenGigabitEthernet2/5/10
description COLT line to THN
no switchport
mtu 8000
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::8:13/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>

C6880-VSS-DC1#show clns protocol

IS-IS Router: 40000
System Id: 4000.4013.4013.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

IS-IS Router: <Null Tag>
System Id: 0000.0000.0000.00 IS-Type: level-1-2
Manual area address(es):
Routing for area address(es):
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: level-1-2
Accept narrow metrics: level-1-2
Generate wide metrics: none
Accept wide metrics: none

IS-IS Router: <ASN>
System Id: 3323.9613.9613.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
Interfaces supported by IS-IS:
TenGigabitEthernet2/5/10 - OSI - IP - IPv6
Port-channel512 - OSI - IP - IPv6
Port-channel510 - OSI - IP - IPv6
Loopback2 - OSI - IP
Loopback1 - OSI - IP - IPv6
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

DC3:
C6880X-VSS-DC3#show run interface te1/5/16
Building configuration...

Current configuration : 277 bytes
!
interface TenGigabitEthernet1/5/16
description COLT line to CHI
no switchport
mtu 8000
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::8:14/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>

C6880X-VSS-DC3#show clns protocol

IS-IS Router: <ASN>
System Id: 3323.9612.9614.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
Interfaces supported by IS-IS:
Vlan2017 - IP
Vlan1105 - OSI - IP - IPv6
TenGigabitEthernet1/5/16 - OSI - IP - IPv6
Port-channel211 - OSI - IP - IPv6
Port-channel210 - OSI - IP - IPv6
Loopback0 - OSI - IP - IPv6
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 100
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

Thank you

[Harold Ritter]

Sorry I meant "Can you provide the output of "show runn interface <lag interface name>" and "show clns proto" from both DC1 and DC2?"

Regards,

[CliveG]

Hi Harold,

Please find below the readout for DC2 and the LAG config for DC1 to DC2 connection on DC1:

DC2:
C6880-VSS-DC2#show run interface port-channel 512
Building configuration...

Current configuration : 292 bytes
!
interface Port-channel512
description Colt to DC1_VSS_TE 1/5/15
no switchport
mtu 9216
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::4:11/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>
end

C6880-VSS-DC2#show clns protocol

IS-IS Router: 40000
System Id: 4000.4011.4011.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

IS-IS Router: MULTICAST
System Id: 3131.3131.3131.00 IS-Type: level-1-2
Manual area address(es):
31
Routing for area address(es):
31
No interfaces in domain/area.
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

IS-IS Router: <ASN>
System Id: 3323.9611.9611.00 IS-Type: level-1-2
Manual area address(es):
49
Routing for area address(es):
49
Interfaces supported by IS-IS:
TenGigabitEthernet1/5/10 - IP
Port-channel512 - OSI - IP - IPv6
Port-channel511 - OSI - IP - IPv6
Loopback0 - IP
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 100
RRR level: none
Generate narrow metrics: none
Accept narrow metrics: none
Generate wide metrics: level-1-2
Accept wide metrics: level-1-2

You will also need the show run for the LAG interface to DC2 from DC1:

DC1:
C6880-VSS-DC1#show run interface port-channel 512
Building configuration...

Current configuration : 268 bytes
!
interface Port-channel512
description To-DC2-LAG
no switchport
mtu 9216
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip pim sparse-dense-mode
ip router isis <ASN>
ipv6 address 2A03:4D80::4:13/112
ipv6 router isis <ASN>
mpls ip
clns router isis <ASN>
end

Many thanks

[Harold Ritter]

Everything looks good so far.

Can you also provide a "sh clns int Port-channel512" from DC1 and DC2.

Also, I see the MTU you set on the port channel interface is 9216. Can you make sure you can ping from DC1 to DC2 to port channel interface ip address using that MTU size ("ping x.x.x.x size 9216 df-bit"). If not, this could be the reason why the ISIS neighbour does not come up.

Regards,

[CliveG]

Hi Harold,

Many thanks for all your help so far.

Here is the requested information:

DC1:
C6880-VSS-DC1#show clns int port-channel 512
Port-channel512 is up, line protocol is up
Checksums enabled, MTU 9213, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
RDPDUs enabled, min. interval 100 msec., Addr Mask enabled
Congestion Experienced bit set at 4 packets
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 32 seconds
Routing Protocol: IS-IS (ASN)
Circuit Type: level-1-2
Interface number 0x3, local circuit ID 0x2
Level-1 Metric: 10, Priority: 64, Circuit ID: C6880-VSS-DC1.02
DR ID: C6880-VSS-CHI.02
Level-1 IPv6 Metric: 10
Number of active level-1 adjacencies: 0
Level-2 Metric: 10, Priority: 64, Circuit ID: C6880-VSS-DC1.02
DR ID: 0000.0000.0000.00
Level-2 IPv6 Metric: 10
Number of active level-2 adjacencies: 0
Next IS-IS LAN Level-1 Hello in 1 seconds
Next IS-IS LAN Level-2 Hello in 3 seconds

DC2:
C6880-VSS-DC2#show clns int port-channel 512
Port-channel512 is up, line protocol is up
Checksums enabled, MTU 9213, Encapsulation SAP
ERPDUs enabled, min. interval 10 msec.
RDPDUs enabled, min. interval 100 msec., Addr Mask enabled
Congestion Experienced bit set at 4 packets
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 20 seconds
Routing Protocol: IS-IS (ASN)
Circuit Type: level-1-2
Interface number 0x2, local circuit ID 0x2
Level-1 Metric: 100, Priority: 64, Circuit ID: C6880-VSS-DC2.02
DR ID: C6880-VSS-THE.02
Level-1 IPv6 Metric: 10
Number of active level-1 adjacencies: 0
Level-2 Metric: 100, Priority: 64, Circuit ID: C6880-VSS-DC2.02
DR ID: 0000.0000.0000.00
Level-2 IPv6 Metric: 10
Number of active level-2 adjacencies: 0
Next IS-IS LAN Level-1 Hello in 377 milliseconds
Next IS-IS LAN Level-2 Hello in 8 seconds

Don't suppose the metrics have anything to do with this? One side is 100 and the other is 10 with no adjacencies showing?

Many thanks Harold. Very much appreciated.