BGP: Why does Cisco router sometimes advertise routes to the peer that it learned the route from?

[jesses5]

I'm testing Cisco Cloud Services Router in a virtualized environment. See the attached image for the topology I've setup. Basically I have 3 CSR's and one vShield Edge that are talking eBGP to each-other.

The problem I'm having is that sometimes, the Cisco CSR will advertise a route back to the peer it learned the route from.

CSR(T) advertises 192.168.1.0/24, and Edge(S) advertises 192.168.0.0/24, so that Source and Target VM can communicate. CSR (1) and (2) sit between CSR(T) and Edge(S) and re-advertise the routes learned.

If CSR(T) is peered ONLY with one of the middle CSR's (1 or 2, but not both), then everything is fine. When I have it peered to both at the same time, then CSR(T) advertises 192.168.0.0/24 back to both CSR(1) and CSR(2). 

CSR(1) is doing something similar - advertising 192.168.1.0/24 to CSR(T) and advertising 192.168.0.0/24 to Edge(S). CSR(2) is configured nearly identically but isn't doing that.

I'm just wondering under what circumstances will a Cisco BGP speaker advertise a route back to the peer it learned it from. It seems like it should never do that. Is there a bug in the Cisco BGP code that I'm running into?

[milan.kulik]

Hi, this question has been discussed here several times in the past https://supportforums.cisco.com/discussion/10875846/bgp-behaviour#3052977 e.g. And the conclusion was: No RFC says: "BGP speaker should not advertise a route back to the peer it learned it from." Cisco routers sometimes don't advertise back and sometimes do. IMHO, it's a question for IOS programmers. Best regards, Milan

[rmulero]

Hi Millan,

i came across this blog. I ran into the same issue. But, the RFC4271 states that you should not advertise with ajd-rib-out. 

But, the statement below could go either way. 

“BGP speaker SHOULD NOT advertise a given feasible BGP route from its Adj-RIB-Out if it would produce an UPDATE message containing the
same BGP route as was previously advertised”

your thoughts. 

Rey

[Joseph Nelson]

Hi,

This may be covered in the thread @Milan referred to, however.

I don't think eBGP has any split horizon mechanism preventing this kind of route reflection ( not like iBGP--don't advertise iBGP learned routes to iBGP peers). So it seems like this behavior is legitimate. Whether CSR(T) advertises the routes back to CSR(1) and CSR(2) isn't really that big of a deal because both CSR(1) and CSR(2) should drop those routes due to AS_PATH loop.

From an administrative perspective, CSR(T) shouldn't announce routes that it hasn't originated. This is an "administrative type" restriction because eBGP peers are allowed to announce routes they don't originate (i.e. eBGP learned routes)--you would like to limit that behavior. You probably want to add an outbound route-map to CSR(T) for both CSR(1) and CSR(2) eBGP peers to limit the announcement to just the 192.168.1.0/24 network.

[jesses5]

I agree that technically this behavior doesn't violate any RFC, however it's clearly wrong behavior even if the RFC doesn't say so.

There will never be a legitimate reason to advertise a route back to the peer that the route was learned from. In fact, given two CSR's eBGP peered to each other, the router receiving this improper advertisement will ignore it because it sees its own AS in the AS-path. If that router didn't ignore the route, then it could create a routing loop.

Also troubling is that this behavior is non-deterministic, given that I have two CSR's that are configured identically (except for having different IP addresses), where one exhibits this wrong behavior while the other doesn't, and then CSR(T) only has this wrong behavior when there's two eBGP peers present.

It would be nice if someone from Cisco could respond with the design intent and explain why this happens, and under what circumstances it happens. If not then it seems like a bug. 

[Jon Marshall]

There will never be a legitimate reason to advertise a route back to the peer that the route was learned from.

Agreed although this is slightly different but the principle still stands. Your router is not strictly advertising a route it learnt from a peer back to that same peer. What it is doing, I think, is advertising a route it learnt from one peer to the other peer and vice versa. The fact that it is actually the same route should stop it from advertising it but that doesn't seem to be the case here.

I'm not arguing against what you are saying, just trying to understand how this behaviour could occur.

So if your router only had one peer then I absolutely wouldn't expect it to advertise back the same route and in fact that is the behaviour you are seeing when you peer with only one neighbour.

The non-deterministic behaviour is a worry. Do you have exactly the same IOS and feature set on both routers ?

I agree it would be nice to hear from someone at Cisco on this.

Jon

[Joseph Nelson]

@Jon Marshall

"Your router is not strictly advertising a route it learnt from a peer back to that same peer. What it is doing, I think, is advertising a route it learnt from one peer to the other peer and vice versa. The fact that it is actually the same route should stop it from advertising it but that doesn't seem to be the case here."

I don't think this applies. In order for CSR(T) to tell CSR(1) about the path through CSR(2), it would have had to select CSR(2) as the best path. If it selected CSR(2) as the best path, by definition, it hasn't selected CSR(1)...ergo CSR(2) would not get the same announcement for a path through CSR(1). BGP will announcing only a single best-path, unless BGP add-path feature is in use.

There will never be a legitimate reason to advertise a route back to the peer that the route was learned from. 

I agree in general this is true. Given that BGP is policy driven, I don't think its a problem BGP needs to solve as you can eliminate this behavior by applying outbound route filter policy. 

That said. I'm interested as well to see Cisco's take on it.

[Jon Marshall]

Joe

I agree entirely with what you are saying.

I was just trying to work out how it could possibly be doing what it is doing and the fact that the same router doesn't do this when it only has one peer made me think it may be something in the code that is allowing it to "reflect" those routes between the peers.

I have probably just muddied the waters with my last post though so i'll shut up now :-)

Jon

[Joseph Nelson]

Jon,

No that's a good line of reasoning and I had the same thought and had to think about it more myself. Although a "show ip bgp nei <ip> advertised-routes" on CSR(T) should reveal that behavior because you'd see CSR(1)s next hop on advertisement to CSR(2) and vice versa.

[jesses5]

Let me clarify-

When CSR(T) is peered to both CSR 1 and 2, it will receive the route for 192.168.0.0/24 from both of them, determine that 172.16.100.100 is the best path (since both paths are equal, the tie-breaker is the lower IP), and advertise a path to 192.168.0.0/24 via 172.16.100.100 (with itself as the next hop) to both CSR1 and CSR2. The advertisement to CSR2 is OK (even though it's undesirable), but the advertisement to CSR1 is not OK since it would create a routing loop if CSR1 didn't ignore it.

The second problem is that CSR1 is advertising the route learned from CSR(T) back to CSR(T) and the same for the route learned from Edge(S). I can't explain why CSR2 doesn't do this. CSR's are virtual routers and were both cloned from the same base template, so they are completely identical in every way except having different interface IPs and different BGP neighbor IPs. The rest of the config is 100% identical.

[milan.kulik]

Hi, what does sh ip bgp update-group display on particular routers? Are both the neighbours within the same group or are they within two separate groups? Best regards, Milan

[jesses5]

I ran it on CSR(T) and it shows both are in the same group-

CSR-Target#show ip bgp update-group
BGP version 4 update-group 15, external, Address Family: IPv4 Unicast
  BGP Update version : 14/0, messages 0
  Route map for outgoing advertisements is BGP
  Topology: global, highest version: 14, tail marker: 14
  Format state: Current working (OK, last minimum advertisement interval)
                Refresh blocked (not in list, last not in list)
  Update messages formatted 6, replicated 9, current 0, refresh 0, limit 1000
  Number of NLRIs in the update sent: max 1, min 0
  Minimum time between advertisement runs is 30 seconds
  Has 2 members:
   172.16.100.100   172.16.101.100

[milan.kulik]

Hi, CSR(2) and CSR(1) output might be interesting then. Best regards, Milan

[jesses5]

CSR1#show ip bgp update-group
BGP version 4 update-group 3, external, Address Family: IPv4 Unicast
  BGP Update version : 7/0, messages 0
  Route map for outgoing advertisements is TO_TARGET
  Topology: global, highest version: 7, tail marker: 7
  Format state: Current working (OK, last minimum advertisement interval)
                Refresh blocked (not in list, last not in list)
  Update messages formatted 7, replicated 7, current 0, refresh 0, limit 1000
  Number of NLRIs in the update sent: max 1, min 0
  Minimum time between advertisement runs is 30 seconds
  Has 1 member:
   172.16.100.1

BGP version 4 update-group 6, external, Address Family: IPv4 Unicast
  BGP Update version : 7/0, messages 0
  Route map for outgoing advertisements is TO_SOURCE
  Topology: global, highest version: 7, tail marker: 7
  Format state: Current working (OK, last minimum advertisement interval)
                Refresh blocked (not in list, last not in list)
  Update messages formatted 0, replicated 0, current 0, refresh 0, limit 1000
  Number of NLRIs in the update sent: max 0, min 0
  Minimum time between advertisement runs is 30 seconds
  Has 1 member:
   172.16.0.1

CSR2#show ip bgp update-group
BGP version 4 update-group 47, external, Address Family: IPv4 Unicast
  BGP Update version : 18/0, messages 0
  Route map for outgoing advertisements is MED_100
  Topology: global, highest version: 18, tail marker: 18
  Format state: Current working (OK, last minimum advertisement interval)
                Refresh blocked (not in list, last not in list)
  Update messages formatted 1, replicated 1, current 0, refresh 0, limit 1000
  Number of NLRIs in the update sent: max 1, min 0
  Minimum time between advertisement runs is 30 seconds
  Has 1 member:
   172.16.101.1

BGP version 4 update-group 48, external, Address Family: IPv4 Unicast
  BGP Update version : 18/0, messages 0
  Topology: global, highest version: 18, tail marker: 18
  Format state: Current working (OK, last minimum advertisement interval)
                Refresh blocked (not in list, last not in list)
  Update messages formatted 1, replicated 1, current 0, refresh 0, limit 1000
  Number of NLRIs in the update sent: max 1, min 0
  Minimum time between advertisement runs is 30 seconds
  Has 1 member:
   172.16.1.1

[milan.kulik]

Hi, I had thought originally a common upgrade-group might be a reason for the behaviour described, But as both CSR(1) and CSR(2) are using two separate upgrade-groups, I was wrong probably. The only difference I see might be the route-maps used but I suppose they are not dejying any prefixes? (Plus there is no route-map towards 172.16.1.1 on CSR(2).) Best regards, Milan