Hi,
I have a recently inherited network that I have a few questions regarding the design and functionality. I have a a diagram attached that I hope will show the design reasonably well.
The Green lines indicate physical link
The Red lines show non-MPLS connections (and also physical links)
The other links that look like spaghetti are iBGP sessions
The network has been setup using MPLS with a BGP free core and using OSPF as the IGP.
The provide an overview, there are 4 sites, 3 of which are fully meshed, the forth (site-D) doesn't have a direct connection to Site-C, therefore, it relies on route reflectors at Site-A and Site-B to provide iBGP routes.
We have recently added edge01 and edge02 in Site-A and will soon add a similar setup to Site-B. The edge devices are ASR9001 routers. As the diagram shows, they have a direct physical link to CE01 and CE02 over which iBGP is run, but not MPLS. There is also a physical connection between edge01 and cor01 (and also edge02 and core02) in which MPLS and iBGP is configured. The purpose of this is to allow traffic to enter the network and reach the other sites without transiting the CE01 or CE02 routers (reducing traffic load).
We have route reflectors setup on CE01 and CE02 at site-A so that the routes from Site-C are seen at Site-D. We have a similar setup at Site-B for the same reason.
I also have another router at Site-B (not in the diagram) that doesn't have a full mesh with any other site, instead, I enabled CE01 and CE02 to reflect the routes to Site-A. When I did this, we saw some routing loops between Edge01 and CE02 at Site-C.
It appeared that CE02 at Site-C did learn a prefix 2 days previous which had since been withdrawn, but edge01 didn't get the update and therefore created a loop. At the time of the looping traffic, edge01 also had a route for the prefix, but it preferred to send it to CE02 at Site-C - which of course didn't have it anymore.
The actual traffic flow went as follows:
CE01-siteA
edge01-SiteA
cor01-SiteA
cor02-SiteA
cor01-Site-C
CE02-site-C (this is where edge01 could see the prefix announcement, but this router no longer had it - loop starts)
cor01-Site-C
cor02-site-A
CE01-Site-A (loop complete - starts over again...)
I'm not at all sure why edge01 still though CE01-SiteC still had a route for the prefix on it's local peering point.
I'm also not clear on why the edge01 router though to send the traffic to CE01-siteC when it also has a local route for it.
What I *think* is occurring, is that for reasons I don't understand, Edge01-SiteA isn't gettting routing updates from the other site's routers, however, routes like CE01-siteA are and are therefore, doing the correct thing and sending to edge01.
So, to my question, is there a fundamental issue with the current setup (I know more use of route reflectors would have been better). Should the Edge devices "talk" MPLS to the CE01 and CE02 devices also?
Is there a better way for the routes on Edge01 and Edge02 to be propagated to Site-B, Site-C and Site-D?
Any idea why Edge01 isn't getting some routing updates - or what I can do to troubleshoot it?
Thanks,
Greg.
