Routing problem with MPLS/ISIS

[CliveG]

Hi,

We have a 3 x DC architecture that can be summarised as per below:

DC1 - West London
DC2 - Telehouse West
DC3 - Telehouse North
DC1 has an ethernet metro line directly to DC2 and DC3 and likewise DC2 and DC3 have a X-Connect within Telehouse, thus forming a tringle.

Originally all of our external traffic from West London (DC1) exited via Ethernet metro to Telehouse North.
To upgrade the connectivity and create DC redundancy, I organised better bandwidth uplinks from Telehouse West.
At DC2 and DC3 as well as transit connectivity we also have LINX Connectivity.
I now have traffic exiting our network via Telehouse West, thus allowing us to upgrade Telehouse North, or should be, except for one issue as explained below:

Traffic destined for the outside world is still traversing Telehouse North (DC3) first and then Telehouse West (DC2) instead of direct to Telehouse West.
Unfortunatly, I am on a network that cannot be "Down" for any reason (other than the obvious complete hardware failure etc).

The internal routing is being handled by ISIS/MPLS and given that I am not a Cisco expert (worked on Juniper mainly over the past several years), I was hoping someone may be able to aid in resolving how I can get the routing changed to go direct over the ethernet metro from DC1 to DC2 without going to DC3 first.

On top of that, I cannot make use of the upgraded LINX connectivity at DC2 because of the above issue. The BGP open packet wants to exit via DC3 and LINX Collector naturally sees this from the wrong address and drops the packets.

Here is the ISIS/MPLS configs I can see on DC1:

router isis 40000
vrf VFM
net 49.4000.4013.4013.00
metric-style wide
redistribute connected

router isis

router isis <ASN>
net 49.3323.9613.9613.00
metric-style wide
redistribute connected
redistribute static ip
maximum-paths 8
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

And DC3:

router isis <ASN>
net 49.3323.9612.9614.00
metric-style wide
metric 100
redistribute connected
redistribute static ip
maximum-paths 8
distance 100 clns
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

And DC2:

router isis 40000
vrf VFM
net 49.4000.4011.4011.00
metric-style wide
redistribute connected

router isis MULTICAST
net 31.3131.3131.3131.00
metric-style wide

router isis <ASN>
net 49.3323.9611.9611.00
metric-style wide
metric 100
redistribute connected
redistribute static ip
maximum-paths 8
distance 100 clns
address-family ipv6
multi-topology
redistribute connected
redistribute static
exit-address-family

If any other information is required then please let me know and I will supply.

Many thanks

[Harold Ritter]

Hi @CliveG ,

> The BGP open packet wants to exit via DC3

Is the BGP peer address a DC2 directly connected prefix?

If so it should use the directly connected link from DC1 to DC2. Is the ISIS neighbor relationship up between DC1 and DC2?

Regards,

[CliveG]

Hi Harold,

So, from what I can see of the neighbor relationships, everything goes to DC3. No wonder everything routes that way. See below neighbor relationships for L1 and L2 IS-IS:

DC1:
DC3 L1 Te2/5/10 xxx.xxx.xxx.xxx UP 8 DC3-THN.03
DC3 L2 Te2/5/10 xxx.xxx.xxx.xxx UP 9 DC3-THN.03

DC2:
DC3 L1 Po511 xxx.xxx.xxx.xxx UP 9 DC3-THN.02
DC3 L2 Po511 xxx.xxx.xxx.xxx UP 7 DC3-THN.02

DC3:
DC2 L1 Po211 xxx.xxx.xxx.xxx UP 26 DC3-THN.02
DC2 L2 Po211 xxx.xxx.xxx.xxx UP 22 DC3-THN.02
DC1 L1 Te1/5/16 xxx.xxx.xxx.xxx UP 27 DC3-THN.03
DC1 L2 Te1/5/16 xxx.xxx.xxx.xxx UP 23 DC3-THN.03

So, the question now would be, how can I get the neighbor relationship up between DC1 and DC2 without affecting any operations of DC3 please?

Many thanks for the help thus far.

[Harold Ritter]

Hi @CliveG ,

You need to configure ISIS on the interface between DC1 and DC2, just like you already do for the interface between DC1 and DC3.

Regards,

[MHM Cisco World]

@Harold Ritter his issue not in bgp.

@CliveG  I remember your old post about connect three DC.

Please share your topolgy here make Mr @Harold Ritter  check it and suggest solution to you.

Thanks  a lot

MHM

[CliveG]

Hi MHM

There is an iBGP relationship between DC1 and DC2 but, as per what I have noted above, the external routes learned are via DC3. As per the discussion with Harold, my guess is because the igp (is-is) has no neighbor relationship between DC1 and DC2.

I am preparing a GNS3 lab to see what happens to the routing when I create an is-is relationship between DC1 and DC2.

I will update when this has been completed.

Thank you both for your help.

[MHM Cisco World]

in BGP you need to see two neighbor in each site 
for example DC1 must see two neighbor DC2 and DC3 
and if the prefix learn from DC2 (ibgp) not direct from DC3(ebgp) that meaning the best path select ibgp over bgp 
this can happened when you LP (inbound or outbound in neighbor) or weight inbound 
MHM

[CliveG]

Hi,

I have just completed a quick check of the ip bgp summary for each DC and it shows as follows:

DC1:
DC2 4 <ASN> 14095246 269514 1783976404 0 0 24w0d 918371
DC3 4 <ASN> 45156394 278323 1783976404 0 0 24w0d 23

DC3:
DC2 4 <ASN> 14249640 45004489 174503500 0 0 24w0d 918357
DC1 4 <ASN> 278325 45156396 174503500 0 0 24w0d 28

DC2:
DC1 4 <ASN> 269517 14095562 241046999 0 0 24w0d 28
DC3 4 <ASN> 45004490 14249803 241046999 0 0 24w0d 28

So, DC1 is getting the full internet routing table from DC2. But here is a read out from the routing tables at DC1 and also the traceroute for the address 8.8.8.8:

Routing Entry for 8.8.8.8 on DC1:
Routing entry for 8.8.8.0/24
Known via

bgp <ASN>

distance 200, metric 2020
Tag 174, type internal
Last update from DC2 1w3d ago
Routing Descriptor Blocks:
* DC2, from DC2, 1w3d ago
Route metric is 2020, traffic share count is 1
AS Hops 3
Route tag 174
MPLS label: none

Traceroute from DC1 to 8.8.8.8:

1 <DC3> [MPLS: Label 50 Exp 0] 0 msec 0 msec 0 msec
2 <DC2 X-Connect port> 0 msec 4 msec 0 msec
3 <Cogent transit ISP>[AS 174] 0 msec 0 msec 4 msec

As you can see with the first hop, there is also an MPLS Label, hence why I titled the subject as MPLS/IS-IS.

Unfortunatly, this network was set up way before I joined the company and looks to me like it was just set up to "work" and then made live with no planning at all. There is zero documentation like HLDs or LLDs that may explain some of it. There was a handover video but the only thing really discussed in that is the sfp's. No help at all.

If there is any more config I can supply then please let me know.

[Harold Ritter]

Hi @CliveG ,

Did you add the configuration to add an ISIS neighbour between DC1 and DC2. Without this, traffic from DC1 to Internet will continue to go through DC3.

Regards,

[CliveG]

Hi Harold,

I cannot do that as of yet because we are running C6880 and I cannot emulate that on GNS3. However, we do have a lab, but I will not be in the DC(1) until tomorrow. I do not want to do this on a live network that cannot afford any downtime without knowing the exact result.

I'll update once I have completed this.

Thank you

[MHM Cisco World]

@Harold Ritter I ask him to draw topolgy to make clear to you I think he is busy 
so let me summary what I know about this issue 
DC1, DC2 and DC3 are connect via L2VPN xconnect and traffic between DC1 and DC2 is pass through DC3
it not take the direct connection between DC1 and DC2 (if there is)
so we must focus in IGP not BGP, in end BGP is peer-to-peer and need reachability that IGP response of. 
hope I am correct @CliveG 
thanks a lot 
MHM

[Harold Ritter]

Hi @MHM Cisco World ,

From the original post:

> DC1 has an ethernet metro line directly to DC2 and DC3 and likewise DC2 and DC3 have a X-

> Connect within Telehouse, thus forming a tringle.

So he just needs to configure ISIS on the direct ethernet link between DC1 and DC2.

Regards,

[CliveG]

Hi Harold,

Now I am a little confused. The LAB that I have is configured exactly the same as the live network but yet at DC1 for the Lab I see the following:

DC2-VSS-01.00-00 0x0000006C 0xFD24 1165 0/0/0
DC2-VSS-01.02-00 0x0000005F 0x6C0E 1102 0/0/0
DC3-VSS-01.00-00 0x0000006C 0x09A5 1028 0/0/0
DC3-VSS-01.04-00 0x0000005C 0x640F 582 0/0/0

And when I traceroute to the dummy internet system it utilises the correct route. Direct from DC1 to DC2. DC3 does not get involved.

Is there anything you would like me to supply you with from the LAB (but the config is the same)?

Thanks

[MHM Cisco World]

I will suggest something here maybe it issue 
you use xconnect and I think the GW of L2 is in DC3 
the BGP use next-hop that reach via GW in DC3
so you need to set next-hop of any prefix in DC1 and DC2 
MHM

[Harold Ritter]

Hi @CliveG ,

I see VSS in the hostname for DC2 and DC3.

Can you provide the output for the following commands from DC2 and DC3. 

show switch virtual
show switch virtual role

Regards,