OSPF equal cost load balancing not working as expected

[emailsbecker]

I've been assigned to clean up our internal corporate LAN/WAN (separate from our production network). My first step is to duplicate our network in GNS3 so I can test the changes I want to make before doing them on the live network but I'm running into problems just getting the initial state of the lab set up. Right now our live network has OSPF equal cost load balancing for some networks. For example:

R4#sh ip rou | b 10.20.0.0
O E2 10.20.0.0/16 [110/20] via 172.25.36.1, 00:45:39, Tunnel40
                  [110/20] via 172.17.199.202, 00:45:39, GigabitEthernet0/1

R4#sh ip rou 10.20.0.0
Routing entry for 10.20.0.0/16
  Known via "ospf 1337", distance 110, metric 20, type extern 2, forward metric 51
  Last update from 172.25.36.1 on Tunnel40, 00:44:56 ago
  Routing Descriptor Blocks:
    172.25.36.1, from 172.25.7.128, 00:44:56 ago, via Tunnel40
      Route metric is 20, traffic share count is 1
  * 172.17.199.202, from 172.25.71.128, 00:44:56 ago, via GigabitEthernet0/1
      Route metric is 20, traffic share count is 1

R4#sh ip ospf data | i 10.20.0.0
10.20.0.0 172.25.7.128 1629 0x80001387 0x009C16 0
10.20.0.0 172.25.71.128 377 0x8000016F 0x00C3C1 0
10.20.0.0 172.25.71.129 740 0x800010F8 0x007D6E 0
10.20.0.0 172.25.127.128 1090 0x800003C0 0x009168 0

I've only built part of the network so far in GNS3 so there are only 2 routers advertising these routes. They do show up in the OSPF database as expected:

GNS3-R4#sh ip ospf data | i 10.20.0.0
10.20.0.0 172.25.7.128 170 0x80000002 0x0063D0 0
10.20.0.0 172.25.71.128 348 0x80000005 0x009B55 0

But I don't get equal cost load balancing despite it looking like it should be working:

GNS3-R4#sh ip rou 10.20.0.0
Routing entry for 10.20.0.0/16
  Known via "ospf 1337", distance 110, metric 20, type extern 2, forward metric 50
  Last update from 172.25.36.1 on Tunnel40, 00:31:51 ago
  Routing Descriptor Blocks:
  * 172.25.36.1, from 172.25.7.128, 00:31:51 ago, via Tunnel40
      Route metric is 20, traffic share count is 1

GNS3-R4#sh ip ospf data ex 10.20.0.0

            OSPF Router with ID (172.17.207.132) (Process ID 1337)

                  Type-5 AS External Link States

  Routing Bit Set on this LSA
  LS age: 426
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 10.20.0.0 (External Network Number )
  Advertising Router: 172.25.7.128
  LS Seq Number: 80000002
  Checksum: 0x63D0
  Length: 36
  Network Mask: /16
       Metric Type: 2 (Larger than any link state path)
       TOS: 0
       Metric: 20
       Forward Address: 0.0.0.0
       External Route Tag: 0

  LS age: 604
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 10.20.0.0 (External Network Number )
  Advertising Router: 172.25.71.128
  LS Seq Number: 80000005
  Checksum: 0x9B55
  Length: 36
  Network Mask: /16
       Metric Type: 2 (Larger than any link state path)
       TOS: 0
       Metric: 20
       Forward Address: 0.0.0.0
       External Route Tag: 0

We don't have maximum-paths configured anywhere in our network, neither in production nor GNS3. I'm not sure what else could be keeping this from working. Any thoughts?

In case it matters, the production device is an 2821 with 12.4(13r)T. I happened to set up GNS3 using a 2691 with 12.4(25d).

[Peter Paluch]

Hello,

It would be great to see how you have connected your topology in GNS because that would answer many questions. Nonetheless, so far, it seems that the router whose OSPF Router ID is 172.25.7.128 is closer to your GNS-R4 than the router whose Router ID is 172.25.71.128. That is most probably why your GNS-R4 chooses only one path.

You can check on the distances from these two ASBRs using the show ip ospf border-routers command.

Best regards,
Peter

[emailsbecker]

Thanks, Peter, I knew there had to be a command for that but couldn't find it. They are the same.

Let me give a little more background. Right now our routing is almost all static routes, and they're in OSPF as E2 routes (which don't properly account for metrics as they're propagated through our network). So we have equal cost load balancing but we shouldn't. What I'm trying to lab up here is to show my boss what'll happen when we change "redistribute static subnets" to "redistribute static subnets metric-type 1". But before I can do that I need to get GNS3 to mirror what we have right now.

I've attached a picture with the topology. The curved colored lines are GRE tunnels built between the sites. SP-R1 and SP-R4 are neighbors. The SM-R1 and OR-R1 devices both have static routes to addresses in the 10.x.x.x range which go into our production network. The SM-00 and OR-00 devices simulate that production network. To keep it simple the SM-00 and OR-00 devices simply have some 10.x.x.x IPs as loopbacks.

Because the ports in GNS3 aren't Gig ports I've manually adjusted the costs of the physical interfaces between SP-R1 and SP-R4 but everything else ought to be the same because they utilize tunnels and think they're directly connected. In both GNS3 and the live network the tunnels all have the same bandwidth configured (2000), no matter what the actual connection's BW is. So on the live network SP-R4 shows this:

i 172.25.71.128 [51] via 172.17.199.202, GigabitEthernet0/1, ASBR, Area 0, SPF 1853
i 172.25.7.128 [50] via 172.25.36.1, Tunnel40, ASBR, Area 0, SPF 1853

In GNS3:

i 172.25.71.128 [51] via 172.17.199.202, FastEthernet0/1, ASBR, Area 0, SPF 7
i 172.25.7.128 [50] via 172.25.36.1, Tunnel40, ASBR, Area 0, SPF 7

Other than the number of the SPF calculation both are the same.

[Peter Paluch]

Hi,

Thanks for the topology!

Well, I guess that the question changes a little - from "why doesn't my GNS3 show multiple paths to 10.20.0.0/16" to "why does my production network show multiple paths to 10.20.0.0/16". Checking the output of show ip ospf border-routers, the 172.25.7.128 is closer than 172.25.71.128, so the redistributed route should be preferred only through the closer one.

Can you please post the output of show ip ospf database external 10.20.0.0 from R4 in your production network? It is quite possible that these LSA-5 in your production network use a non-zero Forwarding Address, and as a result, your R4 in production has two equal cost paths to these Forwarding Addresses. That would explain why your production network uses two paths instead of just one. You would need to reproduce that behavior with your GNS3, then.

Best regards,
Peter

[emailsbecker]

Thanks for the reply Peter.  Sorry for the delay, I was out sick for a few days and then one of my ISP connections was down and the commands you wanted me to run wouldn't have given the info we're looking for.

BTW, one clarification. I mistakenly did the show ip ospf border-routers command once and edited the output to pull out the relevant lines (the production SP-R4 has ~25 lines). This clarifies things a little better:

SP-R4#show ip ospf border-routers | i 172.25.7.128
i 172.25.7.128 [50] via 172.25.36.1, Tunnel40, ASBR, Area 0, SPF 1897

SP-R4#show ip ospf border-routers | i 172.25.71.128
i 172.25.71.128 [51] via 172.17.199.202, GigabitEthernet0/1, ASBR, Area 0, SPF 1897
i 172.25.71.128 [51] via 172.25.36.1, Tunnel40, ASBR, Area 0, SPF 1897

As you mentioned, the costs there are not the same (50 vs 51). The metric for the routes is 20 but the costs there are 50 and 51. I believe that's due to these being E2 routes, which what I'm trying to change but as said, for now I'm just trying to get GNS3 to match our network. As for the other command you requested:

SP-R4#show ip ospf database external 10.20.0.0

OSPF Router with ID (172.17.207.132) (Process ID 1337)

Type-5 AS External Link States

Routing Bit Set on this LSA
LS age: 1350
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 10.20.0.0 (External Network Number )
Advertising Router: 172.25.7.128
LS Seq Number: 8000147E
Checksum: 0xAB0F
Length: 36
Network Mask: /16
Metric Type: 2 (Larger than any link state path)
TOS: 0
Metric: 20
Forward Address: 172.25.7.28
External Route Tag: 0

Routing Bit Set on this LSA
LS age: 1421
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 10.20.0.0 (External Network Number )
Advertising Router: 172.25.71.128
LS Seq Number: 80000266
Checksum: 0xD2BA
Length: 36
Network Mask: /16
Metric Type: 2 (Larger than any link state path)
TOS: 0
Metric: 20
Forward Address: 0.0.0.0
External Route Tag: 0

LS age: 797
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 10.20.0.0 (External Network Number )
Advertising Router: 172.25.71.129
LS Seq Number: 800011EF
Checksum: 0x8C67
Length: 36
Network Mask: /16
Metric Type: 2 (Larger than any link state path)
TOS: 0
Metric: 20
Forward Address: 0.0.0.0
External Route Tag: 0

LS age: 2005
Options: (No TOS-capability, DC)
LS Type: AS External Link
Link State ID: 10.20.0.0 (External Network Number )
Advertising Router: 172.25.127.128
LS Seq Number: 800004B7
Checksum: 0xA061
Length: 36
Network Mask: /16
Metric Type: 2 (Larger than any link state path)
TOS: 0
Metric: 20
Forward Address: 0.0.0.0
External Route Tag: 0

172.25.7.28 (OR-00) is the only non-zero Forward Address (and it's being learned from OR-R1) but all 4 routes have a metric of 20 so I think it would load balance regardless of whether the FA is zero or not. But I'm glad you reminded me about the non-zero FA thing. I've added that to my list of things I need to fix here.

[emailsbecker]

I'm not sure what the policy is here regarding bumping our own threads if they haven't been replied to in a while. If this isn't appropriate let me know.