NickThanks for all the

Nikolaos Milas · ‎05-01-2015

Hello,

On the relatively small-sized organizational network depicted below, we have formed direct connections as follows:

R1, R2 and R3 are running eBGP and iBGP peerings (using Loopback interfaces) over the routes marked red.

(Two of the routers are Cisco 3825 and R1 is 3640.)

All interfaces are Ethernet (Ethernet, FastEthernet or GigabitEthernet).

R1-R2 and R2-R3 Wireless Links are using L2 VLANs to create Point-To-Point links (with respective router subinterfaces for L3 routing). The others are L2 MPLS VPNs (again, with respective router subinterfaces for L3 routing).

Currently, internal routing is being done using static routes (with IP SLA, for failover).

We are planning to setup OSPF so that internal routing among R1, R2 and R3 can be automatically decided (based on link availability).

Now, as we have no experience with OSPF, I would like to ask for some guidance. Assuming equal ospf cost on all above interfaces, a configuration like the following would be sufficient?

R1:

router ospf 120
 network 172.16.1.20 0.0.0.4 area 0
 network 172.16.1.24 0.0.0.4 area 0
 network 172.16.1.30 0.0.0.4 area 0
end

R2:

router ospf 120
 network 172.16.1.16 0.0.0.4 area 0
 network 172.16.1.20 0.0.0.4 area 0
 network 172.16.1.28 0.0.0.4 area 0
 network 172.16.1.30 0.0.0.4 area 0
end

R3:

router ospf 120
 network 172.16.1.16 0.0.0.4 area 0
 network 172.16.1.24 0.0.0.4 area 0
 network 172.16.1.28 0.0.0.4 area 0
end

I think we would not need to specify any interface type (like point-to-multipoint). Is this correct?

Anything else missing from the above config (assuming that all routers will have common default OSPF parameter values)?

Any suggestions, advice about the above concept, design and planned implementation, based on your experience would be highly appreciated.

All the best,
Nick

Jon Marshall · ‎05-01-2015

Nick

You may want to specify "ip ospf network point-to-point" under the interfaces which means there is no need for a DR/BDR election as there are only two routers per subnet so it isn't needed.

But It's not clear what is EBGP and what is IBGP and why you are running them both if you are currently using statics for your internal routing.

In addition your OSPF network statements include the P2P links over MPLS but you are talking about internal routing which I assumed meant using the wireless links ?

It's also not clear which routes you want favoured ie. if you are using statics currently then routing between the sites will use the wireless links assuming you haven't manipulated the static ADs.

If you replace the statics with OSPF then they will take precedence over IBGP routes but not EBGP routes so if you are running EBGP between the routers via the MPLS part traffic will now go that way unless you modify the distance of either EBGP or OSPF.

So we need quite a bit more information to say what will and won't work specifically why run OSPF over all the links and run both EBGP and IBGP as well because I can't see what that gives you, and what path do you want traffic to take if all links are up and running.

Jon

Nikolaos Milas · ‎05-02-2015

Thank you Jon for your reply. I'll try to give answers to your questions:

eBGP: there exist three peerings: R1-to-ISP, R2-to-ISP, R3-to-ISP, over the black lines (connecting R1, R2, R3 to the ISP cloud, respectively). eBGP is used for proper ISP connectivity and allows auto failover from/to the ISP (i.e. the Internet), if some of the 3 peerings to the ISP fails. For example, if connection of R3 to the ISP fails, the traffic from/to the Internet is auto-redirected via R2. [For the L2VPNs, see below, #4.]
iBGP: there exist three peerings (full mesh) between R1, R2, R3 using their Loopback interfaces. iBGP is used (as I understand) to enable continuous BGP routing table sync between R1, R2 and R3.
Internal Routing: Currently, internal routing is static, which means that all LANs behind R1, R2, R3 communicate to one another using static routes:
Traffic between all LANs behind R3 and LANS behind R2 is routed via Int2.4-Int3.4 link. If that link breaks (IP SLA), then it is routed via Intt2.3-Int3.3. Here is how it works:
Traffic between
Primary Link
Failover Link (IP SLA)
R1-R2
Int1.2 - Int2.2
{ Int1.1 – Int2.1 }
R2-R3
Int2.4 - Int3.4
Int2.3 – Int3.3
R1-R3
Int1.5 - Int3.5
{ Int1.2 - Int2.2 - Int2.4 - Int3.4 }
- Red text means that because R1 (3640) does not support IP SLA, we must manually change routes in case Primary Link fails.
- Also note that IPv6 (at least in our routers) does not support IP SLA, so we currently have no auto failover for IPv6 (static) internal routes.
NOTE: The direct links of the 3 routers to the ISP are in fact VLANs (seen as a subinterface on each router). The "P2P" L2VPNs are separate VLANs (encapsulation dot1q, Router-to-Router, using identically numbered subinterfaces) but, naturally, they leverage the same connection to the ISP. If one fails, the other fails too.
These L2VPNs have been implemented (by the ISP) so as to be used like "Tunnels" between our sites, to easily transfer internal (not publicly routable) traffic (rather than using GRE or IPSec tunnels).
The wireless links are also carrying VLANs; each wireless link carries one VLAN for administration (of the many devices, including WiFi and switches forming each link) and one VLAN for data transfer (which appears as "P2P" L2 VLAN). Red lines depict the data transfer VLANs. So, more paths do exist, but data should be carried over these ones; that is the reason we include these particular networks in OSPF.
If we replace static routes with OSPF, we aim in having auto failover of internal routes (including 3640 which does not support IP SLA), by manipulating -if needed- the interface ospf cost.

Now, if we do not specify any static routes (and we do not run OSPF), will iBGP be in a position to find internal routes among R1, R2 and R3? I think not; as someone puts it: "The ultimate rule of thumb for BGP is there must be reachability to the intended peer first before you execute neighbor remote-as command".
(Ref.: http://cyruslab.net/2012/05/22/bgp-injecting-igp-route-into-bgp-without-redistribution/).

Thus, either we should create separate peerings per iBGP neighbor using the actual interfaces (which, for example, translates to eight internal peerings for R2 if we include IPv6), or we should peer using Loopback addresses and we must have already defined routes (static or OSPF-based) to the Loopback addresses...

What would you suggest? The OSPF + iBGP approach seems conceptually more natural to me.

In addition, in either case (iBGP with multiple peerings per neighbor, or OSPF with iBGP - using no sync!), in order to enable all interior routers to exchange routing info for all connected LANs (including private ones), I guess that we should explicitly declare those stub networks (i.e. networks connected to a single router) either to iBGP in the former case or to OSPF in the latter. [In the OSPF case, see red text in the tentative OSPF config below.] Right?

Before closing, here is a modified tentative configuration, following further investigation; this also enables in OSPF the Lo0 interfaces, so that they can be published via OSPF (in order to be used by iBGP):

R1:

router ospf 120
 router-id 1.1.1.1
 log-adjacency-changes
 area 0 authentication message-digest
 
 passive-interface default
 no passive-interface Loopback0
 no passive-interface Int1.1
 no passive-interface Int1.2
 no passive-interface Int1.5

 network 1.1.1.1 0.0.0.0 area 0
 network 172.16.1.20 0.0.0.4 area 0
 network 172.16.1.24 0.0.0.4 area 0
 network 172.16.1.30 0.0.0.4 area 0
 <other stub networks on this router follow here, so that they can be routed internally>
end

ipv6 router ospf 120
 router-id 1.1.1.1
end

R2:

router ospf 120
 router-id 2.2.2.2
 log-adjacency-changes
 area 0 authentication message-digest
 
 passive-interface default
 no passive-interface Loopback0
 no passive-interface Int2.1
 no passive-interface Int2.2
 no passive-interface Int2.3
 no passive-interface Int2.4

 network 2.2.2.2 0.0.0.0 area 0
 network 172.16.1.16 0.0.0.4 area 0
 network 172.16.1.20 0.0.0.4 area 0
 network 172.16.1.28 0.0.0.4 area 0
 network 172.16.1.30 0.0.0.4 area 0
 <other stub networks on this router follow here, so that they can be routed internally>
end

ipv6 router ospf 120
 router-id 2.2.2.2
end

R3:

router ospf 120
 router-id 3.3.3.3
 log-adjacency-changes
 area 0 authentication message-digest
 
 passive-interface default
 no passive-interface Loopback0
 no passive-interface Int3.3
 no passive-interface Int3.4
 no passive-interface Int3.5

 network 3.3.3.3 0.0.0.0 area 0
 network 172.16.1.16 0.0.0.4 area 0
 network 172.16.1.24 0.0.0.4 area 0
 network 172.16.1.28 0.0.0.4 area 0
 <other stub networks on this router follow here, so that they can be routed internally>
end

ipv6 router ospf 120
 router-id 3.3.3.3
end

Additionally, on all interfaces participating in OSPF, if they are IPv6 enabled, I understand that we should add:

ipv6 ospf 120 area 0

(I don't know if there is another way, without needing to add a setting on a per-interface basis.)

As you suggest, we might also need to add to the interface config (on all interfaces participating in OSPF, including Loopback) the setting:

ip ospf network point-to-point

If you find that I am missing or have misunderstood something in logic (conception/design) or in implementation, please direct me.

Thanks a lot for your valuable feedback,
Nick

P.S. Sorry for the multiple edits of this posting, I am trying to make it as meaningful as I can.

Jon Marshall · ‎05-03-2015

Nick

Thanks for all the details.

I need a bit of time to work through it all but I will get back to you.

Couple of last quick questions -

1) I was assuming the ISP setup was an MPLS network but it sounds like it is for internet mainly.

If so does each site have it's own public IP block that is advertised by the local router to the ISP.

And are you receiving full routes from the ISP ?

Sounds like no due to the routers you have so what are you getting from the ISP in terms of routes.

2) IBGP - still trying to work out what is doing here. It sounds like it is making sure each site has alternate paths out to the ISP if their local router connection fails which would make sense.

So again it comes down to what routes are you receiving off the ISP ?

And with those routes do you do any path manipulation eg. some external internet sites must always go via R3 so traffic from all sites is sent that way for specific destinations or does each site always use it's local connection for all destinations if available ?

I am basically trying to work out if it would be easier to simply redistribute EBGP into OSPF with no need for IBGP and let OSPF handle all the internal routing but that is dependant both on how many routes you receives from the ISPs and possibly other things that IBGP is doing in this scenario.

Jon

Nikolaos Milas · ‎05-03-2015

Hi Jon,

Thanks for your reply.

Here follow answers to your questions:

1) I was assuming the ISP setup was an MPLS network but it sounds like it is for internet mainly.

Yes, the ISP is mainly for Internet connectivity. You can see more on the ISP here:

https://www.noc.grnet.gr/en/node/21

and about their L2 & L3 MPLS VPN Services (which is only a small part of the available services) see:

https://www.noc.grnet.gr/en/node/89

If so, does each site have it's own public IP block that is advertised by the local router to the ISP?

Yes, exactly so. In fact, Sites 1, 2, 3 (parts of the same organization) use and advertise 5 Class-C networks and a /48 IPv6 one. The above address spaces are variably subnetted and shared among the 3 sites.

And are you receiving full routes from the ISP?

We receive a table which includes ISP's networks and for the rest we use a default route.

For a more detailed view to our BGP activity, here is the output of "Show IP BGP summary" of R2 (the output for the other routers is almost identical):

#show ip bgp summary
BGP router identifier 195.251.xxx.xxx, local AS number 15690
BGP table version is 23756, main routing table version 23756
1188 network entries using 153252 bytes of memory
4719 path entries using 245388 bytes of memory
36347/193 BGP path/bestpath attribute entries using 4507028 bytes of memory
10135 BGP AS-PATH entries using 290036 bytes of memory
1649 BGP community entries using 141882 bytes of memory
14 BGP extended community entries using 352 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5337938 total bytes of memory
1177 received paths for inbound soft reconfiguration
BGP activity 99119/75539 prefixes, 1014203/942312 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
62.217.xxx.xxx  4  5408  312269  298439    23756    0    0 3w6d         1177
195.251.xxx.yyy 4 15690   20026   20055    23756    0    0 2d06h        1179
195.251.xxx.zzz 4 15690   20303   20333    23756    0    0 2d07h        1179

eBGP peering in red, iBGP peerings in blue.

IPv6 eBGP on the same router:

#show bgp ipv6 unicast summary
BGP router identifier 195.251.xxx.xxx, local AS number 15690
BGP table version is 1947748, main routing table version 1947748
22392 network entries using 3605112 bytes of memory
67172 path entries using 5105072 bytes of memory
36347/12132 BGP path/bestpath attribute entries using 4507028 bytes of memory
10135 BGP AS-PATH entries using 290036 bytes of memory
1649 BGP community entries using 141882 bytes of memory
14 BGP extended community entries using 352 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 13649482 total bytes of memory
22390 received paths for inbound soft reconfiguration
BGP activity 99125/75545 prefixes, 1014566/942675 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2001:648:2011:xxx::1
                4 15690   76614   86624  1947748    0    0 2d07h       22391
2001:648:2FFD:xxx:xxx::1
                4  5408 1183021  298453  1947739    0    0 4w6d        22390

Note that there is only one iBGP peering for IPv6 since R1 does not support IPv6 (but it will be replaced soon).

2) IBGP - still trying to work out what is doing here. It sounds like it is making sure each site has alternate paths out to the ISP if their local router connection fails which would make sense.

Yes, this is the case.

So again it comes down to what routes are you receiving off the ISP ?

Yes.

And with those routes do you do any path manipulation eg. some external internet sites must always go via R3 so traffic from all sites is sent that way for specific destinations or does each site always use it's local connection for all destinations if available ?

The latter is the case; yet we would like to change this scheme in the future, so that R3's traffic from/to the ISP/Internet goes through R2 always (unless of course connection to R2 is down). Studying for it, I believe we can achieve this by appropriately changing the current values of metric (to affect ingress traffic) and local-preference (to affect egress traffic) in the BGP configuration of R1, R2 and R3. (General guidelines about BGP policies in our ISP's AS, which we follow as well: http://www.onesc.net/communities/as5408/)

I hope the above info provides the details you need.

All the best,
Nick

Jon Marshall · ‎05-05-2015

Nick

Apologies for the delay in getting back on this.

I think the answer is basically yes you can run OSPF as you proposed across both links per site and then you would need to manipulate the interface costs at each site to make the traffic flow the way you wanted if both links were up.

I was considering removing IBGP and redistributing EBGP into OSPF but as you want to use IBGP to influence which site is used for certain networks in the future you probably want to keep that and just OSPF for your internal routes only.

I did also consider using areas for OSPF ie. your wireless links were in one area and your ISP VPNs in another but as they both terminate on the same routers I can't see any advantage to that and it just complicates things.

So yes, OSPF across all links and you would need to modify the cost to make the wireless links favoured if both are up.

If you have more questions etc. then please come back.

Jon

Nikolaos Milas · ‎05-05-2015

Thank you Jon for your review of the situation.

In terms of implementation -I have no experience with OSPF-, would you think the tentative configuration above is OK?

Maybe it misses any critical part(s)? Any comments on it?

Thanks again,
Nick

Jon Marshall · ‎05-05-2015

Nick

I don't really have any experience with IPv6 so can't really comment on that.

For the rest of your configuration -

1) you are setting the router id and then using a network statement for it but that won't advertise it unless you have also assigned that IP to an interface.

Have you also assigned the same IP to a loopback interface on the router ?

If so then it will be advertised but you don't then need to set it explicitly with the router id as far as I know unless you have other loopbacks with a higher IP configured on them.

To be honest I have never tried using a loopback with the IP and using the same IP as the router id.

2) your wildcard mask for your OSPF network statements is wrong. It should be 0.0.0.3

3) you have configured OSPF authentication. Presumably you are going to configure the key under each interface ?

Jon

Nikolaos Milas · ‎05-06-2015

Thank you Jon,

You have been most helpful.

On 1: Yes, we have a single Lo0 interface on all routers with an IP Address. So, per your advice, I will not configure an explicit router id for OSPF.

On 2: You are quite right! My bad.

On 3. Yes, we plan to configure a common key under all OSPF-enabled interfaces:

ip ospf message-digest-key 10 md5 oursecretkey

Thanks again,
Nick

Traffic between	Primary Link	Failover Link (IP SLA)
R1-R2	Int1.2 - Int2.2	{ Int1.1 – Int2.1 }
R2-R3	Int2.4 - Int3.4	Int2.3 – Int3.3
R1-R3	Int1.5 - Int3.5	{ Int1.2 - Int2.2 - Int2.4 - Int3.4 }

OSPF Configuration over P2P VLANs