How is my traffic being distributed/load-balanced here across two WAN links? No BGP metrics.

[colossus1611]

Hi All,

Just came across a new small customer site and can't understand how is the traffic being distributed/load-balanced across two WAN links on two separate WAN routers here. The configuration doesn't seem to have any BGP metrics to influence this. I am suspecting CEF load-balancing, but get no evidence of it (or possibly I don't know the right commands to look for it). The users are complaining of Voice traffic drop-outs and I have observed that this happens when the secondary link has high latency on it (something that I test with point to point pings with higher packet size). I would like to eliminate this secondary link as an issue here and hence firstly would like to know why am I seeing traffic on this secondary WAN link when primary is UP on first router all the time. The primary WAN link on RTR01 is 30mbps and secondary WAN link on RTR02 is 20mbps.

Here's some show outputs from both these routers:

RTR01#show int des
Interface Status Protocol Description
Gi0/0 up up Primary WAN link
Gi0/1 up up To LAN Distribution switch
Gi0/2 up up To LAN Distribution switch

RTR02#show int des
Interface Status Protocol Description
Gi0/0 up up Secondary WAN link
Gi0/1 up up To LAN Distribution Switch
Gi0/2 up up To LAN Distribution Switch

RTR02#show int gi0/0
Internet address is 10.252.0.42/30
MTU 1500 bytes, BW 20000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Last input 00:00:04, output 00:00:00, output hang never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3397863
Queueing strategy: Class-based queueing
Output queue: 0/1000/3397834 (size/max total/drops)
1 minute input rate 63000 bits/sec, 20 packets/sec
1 minute output rate 35000 bits/sec, 12 packets/sec

RTR01#show int gi0/0
Internet address is 10.252.0.54/30
MTU 1500 bytes, BW 30000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Full Duplex, 100Mbps, media type is RJ45
Last input 00:00:00, output 00:00:00, output hang never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1054415
Queueing strategy: Class-based queueing
Output queue: 0/1000/1054257 (size/max total/drops)
1 minute input rate 72000 bits/sec, 17 packets/sec
1 minute output rate 94000 bits/sec, 32 packets/sec
1563558175 packets input, 963665363 bytes, 0 no buffer
Received 5165957 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
2763543459 packets output, 3923617842 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
5165956 unknown protocol drops

RTR01#show run | s bgp
router bgp 12345
bgp log-neighbor-changes
redistribute connected
redistribute ospf 100 metric 10 route-map ospf-to-bgp
neighbor 10.252.0.53 remote-as 6789
neighbor 10.252.0.53 update-source GigabitEthernet0/0
neighbor 10.252.0.53 timers 8 24
neighbor 10.252.0.53 soft-reconfiguration inbound
neighbor 10.252.0.53 route-map bgp-in in
neighbor 172.31.232.3 remote-as 12345
neighbor 172.31.232.3 update-source Port-channel1.10
neighbor 172.31.232.3 soft-reconfiguration inbound
route-map bgp-in deny 10
match ip address prefix-list ospf-in
route-map bgp-in permit 20
route-map ospf-to-bgp permit 10
match ip address prefix-list ospf-in
set origin igp
snmp-server enable traps bgp

RTR01#show ip prefix-list ospf-in
ip prefix-list ospf-in: 1 entries
seq 10 permit 172.31.232.0/22 le 32

RTR01#show route-map bgp-in
route-map bgp-in, deny, sequence 10
Match clauses:
ip address prefix-lists: ospf-in
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map bgp-in, permit, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes

RTR02#show run | s bgp
router bgp 12345
bgp log-neighbor-changes
redistribute connected
redistribute ospf 100 metric 10 route-map ospf-to-bgp
neighbor 10.252.0.41 remote-as 6789
neighbor 10.252.0.41 update-source GigabitEthernet0/0
neighbor 10.252.0.41 soft-reconfiguration inbound
neighbor 10.252.0.41 route-map bgp-in in
neighbor 172.31.232.2 remote-as 12345
neighbor 172.31.232.2 update-source Port-channel2.10
neighbor 172.31.232.2 soft-reconfiguration inbound
route-map bgp-in deny 10
match ip address prefix-list ospf-in
route-map bgp-in permit 20
route-map ospf-to-bgp permit 10
match ip address prefix-list ospf-in
set origin igp
snmp-server enable traps bgp

RTR02#show route-map bgp-in
route-map bgp-in, deny, sequence 10
Match clauses:
ip address prefix-lists: ospf-in
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map bgp-in, permit, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes

RTR02#show ip prefix-list ospf-in
ip prefix-list ospf-in: 1 entries
seq 10 permit 172.31.232.0/22 le 32

[Georg Pauwen]

Hello,

post  a schematic drawing of the entire topology, as well as the full running configurations of the two routers (sh run).

[colossus1611]

Hi Georg,

Please find attached the show run output for both WAN routers (Edit - let me know pls if the attachments aren't visible, as I can't see it for some reason after I posted). 

The topology is as below:

[colossus1611]

Posting truncated config text for RTR01 and RTR02 as attachments aren't visible  (atleast to me).

hostname RTR01
!
no ip source-route
!
ip cef
no ipv6 cef
!
track 1 interface GigabitEthernet0/0 line-protocol
!
class-map match-any MEDIUM-PRIORITY
match access-group name SAP
match protocol ldap
match protocol ipsec
match ip dscp cs2
match ip dscp af21
class-map match-any WAN-VOICE
match ip dscp ef
class-map match-any WAN-VIDEO
match ip dscp af41
class-map match-any HIGH-PRIORITY
match protocol telnet
match protocol snmp
match protocol dns
match protocol rtcp
match protocol ssh
match ip dscp cs3
match ip dscp af31
class-map match-any WAN-MED
match ip dscp af21
class-map match-any VIDEO
match ip dscp cs4
match ip dscp af41
class-map match-any VOICE
match ip dscp ef
class-map match-any WAN-HIGH
match ip dscp af31
!
policy-map WAN-QOS
class WAN-VOICE
priority percent 30
class WAN-HIGH
bandwidth percent 10
class WAN-MED
bandwidth percent 10
class WAN-VIDEO
bandwidth percent 40
class class-default
fair-queue
random-detect dscp-based
set dscp default
policy-map WAN-SHAPING
class class-default
shape average 30000000
service-policy WAN-QOS
policy-map IN-QOS
class VOICE
set dscp ef
class HIGH-PRIORITY
set dscp af31
class MEDIUM-PRIORITY
set dscp af21
class VIDEO
set dscp af41
!
interface Loopback0
ip address 172.31.1.21 255.255.255.255
!
interface Port-channel1
description (M)SWD01:Po1
no ip address
no ip proxy-arp
load-interval 60
service-policy input IN-QOS
hold-queue 150 in
!
interface Port-channel1.10
encapsulation dot1Q 10
ip address 172.31.232.2 255.255.255.128
no ip proxy-arp
standby 1 ip 172.31.232.1
standby 1 priority 110
standby 1 preempt
standby 1 track 1 decrement 20
ip ospf 100 area 0.0.0.0
!
interface GigabitEthernet0/0
description Primary WAN link
bandwidth 30000
ip address 10.252.0.54 255.255.255.252
no ip proxy-arp
ip flow ingress
ip flow egress
load-interval 60
duplex full
speed 100
no cdp enable
service-policy output WAN-SHAPING
!
interface GigabitEthernet0/1
description (M)SWD01:Gi1/0/1
no ip address
no ip proxy-arp
duplex auto
speed auto
channel-group 1
!
interface GigabitEthernet0/2
description (M)SWD01:Gi2/0/1
no ip address
no ip proxy-arp
duplex auto
speed auto
channel-group 1
!
router ospf 100
default-information originate
distribute-list prefix ospf-in in
!
router bgp 12345
bgp log-neighbor-changes
redistribute connected
redistribute ospf 100 metric 10 route-map ospf-to-bgp
neighbor 10.252.0.53 remote-as 6789
neighbor 10.252.0.53 update-source GigabitEthernet0/0
neighbor 10.252.0.53 timers 8 24
neighbor 10.252.0.53 soft-reconfiguration inbound
neighbor 10.252.0.53 route-map bgp-in in
neighbor 172.31.232.3 remote-as 12345
neighbor 172.31.232.3 update-source Port-channel1.10
neighbor 172.31.232.3 soft-reconfiguration inbound
!
ip flow-top-talkers
top 20
sort-by bytes
cache-timeout 900000
!
ip prefix-list ospf-in seq 10 permit 172.31.232.0/22 le 32
ip sla 21
udp-jitter 172.31.1.251 20000 codec g711alaw
ip sla schedule 21 life forever start-time now
ip sla 22
icmp-echo 172.31.1.251
ip sla schedule 22 life forever start-time now
ip sla responder
!
route-map bgp-in deny 10
match ip address prefix-list ospf-in
!
route-map bgp-in permit 20
!
route-map ospf-to-bgp permit 10
match ip address prefix-list ospf-in
set origin igp
!
end

hostname RTR02

no ip source-route
!
ip cef
no ipv6 cef
redundancy
!
class-map match-any MEDIUM-PRIORITY
match access-group name SAP
match protocol ldap
match protocol ipsec
match ip dscp cs2
match ip dscp af21
class-map match-any WAN-VOICE
match ip dscp ef
class-map match-any WAN-VIDEO
match ip dscp af41
class-map match-any HIGH-PRIORITY
match protocol telnet
match protocol snmp
match protocol dns
match protocol rtcp
match protocol ssh
match ip dscp cs3
match ip dscp af31
class-map match-any WAN-MED
match ip dscp af21
class-map match-any VIDEO
match ip dscp cs4
match ip dscp af41
class-map match-any VOICE
match ip dscp ef
class-map match-any WAN-HIGH
match ip dscp af31
!
policy-map WAN-QOS
class WAN-VOICE
priority percent 30
class WAN-HIGH
bandwidth percent 10
class WAN-MED
bandwidth percent 10
class WAN-VIDEO
bandwidth percent 40
class class-default
fair-queue
random-detect dscp-based
set dscp default
policy-map WAN-SHAPING
class class-default
shape average 20000000
service-policy WAN-QOS
policy-map IN-QOS
class VOICE
set dscp ef
class HIGH-PRIORITY
set dscp af31
class MEDIUM-PRIORITY
set dscp af21
class VIDEO
set dscp af41
!
interface Loopback0
ip address 172.31.1.24 255.255.255.255
!
interface Port-channel2
description (M)SWD01:Po2
no ip address
no ip proxy-arp
load-interval 60
service-policy input IN-QOS
hold-queue 150 in
!
interface Port-channel2.10
encapsulation dot1Q 10
ip address 172.31.232.3 255.255.255.128
no ip proxy-arp
standby 1 ip 172.31.232.1
standby 1 preempt
ip ospf 100 area 0.0.0.0
!
interface GigabitEthernet0/0
description Secondary WAN link
bandwidth 20000
ip address 10.252.0.42 255.255.255.252
no ip proxy-arp
ip flow ingress
ip flow egress
load-interval 60
duplex auto
speed auto
no cdp enable
service-policy output WAN-SHAPING
!
interface GigabitEthernet0/1
description (M)SWD01:Gi1/0/2
no ip address
no ip proxy-arp
duplex auto
speed auto
channel-group 2
!
interface GigabitEthernet0/2
description (M)SWD01:Gi2/0/2
no ip address
no ip proxy-arp
duplex auto
speed auto
channel-group 2
!
router ospf 100
default-information originate
distribute-list prefix ospf-in in
!
router bgp 12345
bgp log-neighbor-changes
redistribute connected
redistribute ospf 100 metric 10 route-map ospf-to-bgp
neighbor 10.252.0.41 remote-as 6789
neighbor 10.252.0.41 update-source GigabitEthernet0/0
neighbor 10.252.0.41 soft-reconfiguration inbound
neighbor 10.252.0.41 route-map bgp-in in
neighbor 172.31.232.2 remote-as 12345
neighbor 172.31.232.2 update-source Port-channel2.10
neighbor 172.31.232.2 soft-reconfiguration inbound
!
ip flow-top-talkers
top 20
sort-by bytes
cache-timeout 900000
!
!
ip access-list standard ALL-ROUTES-OUT-v01
permit any

ip access-list extended SAP
permit ip any 10.38.64.0 0.0.1.255
!
ip prefix-list ospf-in seq 10 permit 172.31.232.0/22 le 32
ip sla 21
udp-jitter 172.31.1.251 20000 codec g711alaw
ip sla schedule 21 life forever start-time now
ip sla 22
icmp-echo 172.31.1.251
ip sla schedule 22 life forever start-time now
ip sla responder
logging source-interface Loopback0
!
route-map bgp-in deny 10
match ip address prefix-list ospf-in
!
route-map bgp-in permit 20
!
route-map ospf-to-bgp permit 10
match ip address prefix-list ospf-in
set origin igp
!
end

[Giuseppe Larosa · Accepted solution]

Hello @colossus1611 ,

you can have per flow load balancing in the upstream direction (to the internet) if the distribution switches participate in OSPF and they see two default route in OSPF injected by R1 and R2.

This would not be true if they would use a default static route pointing to the HSRP VIP in po1.10 ( this may be a relic the trace of an old configuration).

For traffic coming from the internet to your prefixes how traffic come back depend on the topology within ISP AS 6789. Traffic can come back via a single path if both PE routers have the same IGP cost from core but one of them has a lower BGP RID.

There is little you can do about this just to ask to the ISP to implement multpath in order to have both WAN links used in downstream.

To check how many prefixes you are learning you can use

show ip bgp summary

the righmost column provides the number of prefixes received

Hope to help

Giuseppe

[colossus1611]

Hi Giuseppe,

Yes that's absolutely right. The distribution swtich is participating in OSPF process and has two equal cost routes towards RTR01 and RTR02.

SWD01#

O*E2 0.0.0.0/0 [110/1] via 172.31.232.3, 6d14h, Vlan10
[110/1] via 172.31.232.2, 6d14h, Vlan10

As for the return routes, I wasn't sure if there's anything that the rotuer has on it's configruation to do cef load balancing (I couldnt' find any), so thanks for confirming that this is definitely being managed by ISP and not us. The second ISP link is showing BPG drops and output packets drops, for which they aren't taking responsibility.

[Leonid Voronkin]

Does AS6789 advertise to you only default through both uplinks? Or maybe full-view?

Can you show show ip bgp from both routers? In case of full-view don't show all output. Just tell about it :-)

Plus show ip bgp neighbors 10.252.0.53 advertised-routes from RTR01 and show ip bgp neighbors 10.252.0.41 advertised-routes from RTR02

[colossus1611]

Hi Leonid,

Yes I am getting more than default routes from AS6789.

Here's the routes being advertised, but nothing fancy there. I think as Giuseppe confirmed this is definitely being load balanced at distribution switch for outgoing routes. It's the incoming traffic that I was wondering about and how it is load balanced as I couldn't see any traces of BGP or CEF load balancing configuration on routers. Seems like ISP might be controlling it inbound.

RTR01#show ip bgp neighbors 10.252.0.53 advertised-routes
Network Next Hop Metric LocPrf Weight Path
*>i 10.252.0.40/30 172.31.232.3 0 100 0 ?
*> 10.252.0.52/30 0.0.0.0 0 32768 ?
*>i 10.252.0.56/30 10.252.0.41 0 100 0 6789 i
*> 172.31.1.21/32 0.0.0.0 0 32768 ?
*>i 172.31.1.24/32 172.31.232.3 0 100 0 ?
*> 172.31.232.0/25 0.0.0.0 0 32768 ?
*> 172.31.232.128/25
172.31.232.5 10 32768 i
*> 172.31.233.0/24 172.31.232.5 10 32768 i
*> 172.31.234.0/24 172.31.232.5 10 32768 i
*> 172.31.235.0/24 172.31.232.5 10 32768 i

RTR02#show ip bgp neighbors 10.252.0.41 advertised-routes
Network Next Hop Metric LocPrf Weight Path
*>i 10.252.0.8/30 10.252.0.53 0 100 0 6789 i
*> 10.252.0.40/30 0.0.0.0 0 32768 ?
*>i 10.252.0.52/30 172.31.232.2 0 100 0 ?
*>i 10.252.0.76/30 10.252.0.53 0 100 0 6789 i
*>i 10.252.0.112/30 10.252.0.53 0 100 0 6789 i
*>i 172.31.1.21/32 172.31.232.2 0 100 0 ?
*>i 172.31.1.23/32 10.252.0.53 0 100 0 6789 i
*> 172.31.1.24/32 0.0.0.0 0 32768 ?
*> 172.31.232.0/25 0.0.0.0 0 32768 ?
*> 172.31.232.128/25
172.31.232.5 10 32768 i
*> 172.31.233.0/24 172.31.232.5 10 32768 i
*> 172.31.234.0/24 172.31.232.5 10 32768 i
*> 172.31.235.0/24 172.31.232.5 10 32768 i
*>i 172.31.245.0/25 10.252.0.53 0 100 0 6789 i
*>i 172.31.245.128/25
10.252.0.53 0 100 0 6789 i