BGP troubleshooting assistance

bberry · ‎03-05-2014

Hi all,

I currently have a 2921 running 15.1(4)M4 code. I have a 2xT1 (3M) serial link with my current ISP. This link is using BGP to share routes with my ISP and the rest of my remote locations. I have not had any issues with this other than bandwidth so we ordered a new 20M Ethernet link with the same provider. They finally delivered the new link last week and I had the turn up with the provider on Monday. That is when I started noticing odd things about the new link.

I am able to ping the new peer over the Ethernet link so I added the peer to my BGP configuration. I expected to start seeing either double routes advertised in my routing table or new entries preferring the new 20M Ethernet link. All that I see is the existing routing table with routes pointing to my existing 3M link. I am able to pull up the BGP peer information and I am seeing numbers in regards to prefixs and things that look close. One item I have noticed is that on the 3M peer I have more Rcvd Prefix activity but do not know if this is something on the provider side or something missing on my side. I have a default static route defined pointing to the 3M peer that we started using when we had BPG issues in the past but I have tried pulling that out as well as changing it to my new 20M peer. All I get is the same routing table with 3M destinations and users complaining about slower than normal performance.

This is the first time I have ever had to deal with two disperate links like this in the same router. What is my next step in trying to find the root cause of this issue? I will admit BGP is a rather week area for me. This is my current route configuration BGP nei informtion as well as a snipit of my routing table ...

*** Routing table ***

PH_BldgA_r#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 10.1.4.58 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 10.1.4.58
                [1/0] via 10.1.4.46
      10.0.0.0/8 is variably subnetted, 96 subnets, 6 masks
B        10.1.1.132/30 [20/0] via 10.1.4.58, 1d17h
B        10.1.2.17/32 [20/0] via 10.1.4.46, 22:54:51
B        10.1.2.25/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.29/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.33/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.37/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.41/32 [20/0] via 10.1.4.46, 1d17h
C        10.1.2.45/32 is directly connected, Loopback0
B        10.1.2.49/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.53/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.61/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.73/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.77/32 [20/0] via 10.1.4.46, 1d17h
B        10.1.2.81/32 [20/0] via 10.1.4.46, 1d17h

*** Roting config ***

router eigrp 10

default-metric 10000 100 255 1 1500

network 10.0.0.0

network 172.17.0.0

redistribute static

redistribute bgp 65006

!

router bgp 65006

bgp log-neighbor-changes

bgp redistribute-internal

redistribute connected metric 100

redistribute eigrp 10 metric 100

neighbor 10.1.4.46 remote-as 13979

neighbor 10.1.4.46 soft-reconfiguration inbound

neighbor 10.1.4.58 remote-as 13979

neighbor 10.1.4.58 soft-reconfiguration inbound

!router eigrp 10
default-metric 10000 100 255 1 1500
network 10.0.0.0
network 172.17.0.0
redistribute static
redistribute bgp 65006
!
router bgp 65006
bgp log-neighbor-changes
bgp redistribute-internal
redistribute connected metric 100
redistribute eigrp 10 metric 100
neighbor 10.1.4.46 remote-as 13979
neighbor 10.1.4.46 soft-reconfiguration inbound
neighbor 10.1.4.58 remote-as 13979
neighbor 10.1.4.58 soft-reconfiguration inbound
!

*** ip bgp nei ***

BGP neighbor is 10.1.4.46, remote AS 13979, external link
BGP version 4, remote router ID 12.123.67.172
BGP state = Established, up for 8w5d
Last read 00:00:14, last write 00:00:34, hold time is 180, keepalive interval is 60 seconds
Neighbor sessions:
    1 active, is not multisession capable (disabled)
Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised and received
    Address family IPv4 Unicast: advertised and received
    Graceful Restart Capability: received
      Remote Restart timer is 120 seconds
      Address families advertised by peer:
        IPv4 Unicast (was not preserved)
    Multisession Capability:
Message statistics:
    InQ depth is 0
    OutQ depth is 0

                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:             1757       7844
    Keepalives:         97064      87351
    Route Refresh:          0          0
    Total:              98822      95196
Default minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast
Session: 10.1.4.46
BGP table version 267067, neighbor version 267067/0
Output queue size : 0
Index 19, Advertise bit 0
19 update-group member
Inbound soft reconfiguration allowed
AF-dependant capabilities:
    Outbound Route Filter (ORF) type (128) Prefix-list:
      Receive-mode: received
Slow-peer detection is disabled
Slow-peer split-update-group dynamic is disabled
                                 Sent       Rcvd
Prefix activity:               ----       ----
    Prefixes Current:             290        254 (Consumes 14224 bytes)
    Prefixes Total:              6507      16565
    Implicit Withdraw:            327       1153
    Explicit Withdraw:           5890      15158
    Used as bestpath:             n/a        252
    Used as multipath:            n/a          0

                                   Outbound    Inbound
Local Policy Denied Prefixes:    --------    -------
    AS_PATH loop:                       n/a       9202
    Bestpath from this peer:           5081        n/a
    Invalid Path:                     14866        n/a
    Total:                            19947       9202
Number of NLRIs in the update sent: max 301, min 0
Last detected as dynamic slow peer: never
Dynamic slow peer recovered: never

Address tracking is enabled, the RIB does have a route to 10.1.4.46
Connections established 9; dropped 8
Last reset 8w5d, due to Peer closed the session of session 1
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1
Local host: 10.1.4.45, Local port: 46876
Foreign host: 10.1.4.46, Foreign port: 179
Connection tableid (VRF): 0
Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0xB23B4B9C8):
Timer          Starts    Wakeups            Next
Retrans         98699          1             0x0
TimeWait            0          0             0x0
AckHold         92104      90239             0x0
SendWnd             0          0             0x0
KeepAlive           0          0             0x0
GiveUp              0          0             0x0
PmtuAger      5311033    5311032     0xB23B4BC1B
DeadWait            0          0             0x0
Linger              0          0             0x0
ProcessQ            0          0             0x0

iss: 2967784113 snduna: 2969749024 sndnxt: 2969749024
irs: 3362379524 rcvnxt: 3364539541

sndwnd: 32768 scale: 0 maxrcvwnd: 16384
rcvwnd: 15282 scale: 0 delrcvwnd: 1102

SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 4 ms, maxRTT: 1000 ms, ACK hold: 200 ms
Status Flags: active open
Option Flags: nagle, path mtu capable
IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):
Rcvd: 189771 (out of order: 0), with data: 92105, total data bytes: 2160016
Sent: 191112 (retransmit: 1, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 98702, total data bytes: 1964910
Packets received in fast path: 0, fast processed: 0, slow path: 0
fast lock acquisition failures: 0, slow path: 0

BGP neighbor is 10.1.4.58, remote AS 13979, external link
BGP version 4, remote router ID 12.123.71.162
BGP state = Established, up for 1d20h
Last read 00:00:11, last write 00:00:40, hold time is 180, keepalive interval is 60 seconds
Neighbor sessions:
    1 active, is not multisession capable (disabled)
Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised and received
    Address family IPv4 Unicast: advertised and received
    Graceful Restart Capability: received
      Remote Restart timer is 120 seconds
      Address families advertised by peer:
        IPv4 Unicast (was preserved)
    Multisession Capability:
Message statistics:
    InQ depth is 0
    OutQ depth is 0

                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:               74          2
    Keepalives:          2917       2664
    Route Refresh:          0          0
    Total:               2992       2667
Default minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast
Session: 10.1.4.58
BGP table version 267067, neighbor version 267067/0
Output queue size : 0
Index 19, Advertise bit 0
19 update-group member
Inbound soft reconfiguration allowed
AF-dependant capabilities:
    Outbound Route Filter (ORF) type (128) Prefix-list:
      Receive-mode: received
Slow-peer detection is disabled
Slow-peer split-update-group dynamic is disabled
                                 Sent       Rcvd
Prefix activity:               ----       ----
    Prefixes Current:             290          2 (Consumes 112 bytes)
    Prefixes Total:              6507          2
    Implicit Withdraw:            327          0
    Explicit Withdraw:           5890          0
    Used as bestpath:             n/a          1
    Used as multipath:            n/a          0

                                   Outbound    Inbound
Local Policy Denied Prefixes:    --------    -------
    Bestpath from this peer:           5081        n/a
    Invalid Path:                     14866        n/a
    Total:                            19947          0
Number of NLRIs in the update sent: max 37, min 0
Last detected as dynamic slow peer: never
Dynamic slow peer recovered: never

Address tracking is enabled, the RIB does have a route to 10.1.4.58
Connections established 2; dropped 1
Last reset 1d20h, due to Peer closed the session
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1
Local host: 10.1.4.57, Local port: 51517
Foreign host: 10.1.4.58, Foreign port: 179
Connection tableid (VRF): 0
Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0xB23B4B9CC):
Timer          Starts    Wakeups            Next
Retrans          2953          1             0x0
TimeWait            0          0             0x0
AckHold          2666       2610             0x0
SendWnd             0          0             0x0
KeepAlive           0          0             0x0
GiveUp              0          0             0x0
PmtuAger       158204     158203     0xB23B4BC1B
DeadWait            0          0             0x0
Linger              0          0             0x0
ProcessQ            0          0             0x0

iss: 3726432152 snduna: 3726494168 sndnxt: 3726494168
irs: 3944421558 rcvnxt: 3944472352

sndwnd: 31761 scale: 0 maxrcvwnd: 16384
rcvwnd: 15339 scale: 0 delrcvwnd: 1045

SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 4 ms, maxRTT: 1000 ms, ACK hold: 200 ms
Status Flags: active open
Option Flags: nagle, path mtu capable
IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):
Rcvd: 5630 (out of order: 0), with data: 2665, total data bytes: 50793
Sent: 5621 (retransmit: 1, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 2954, total data bytes: 62015
Packets received in fast path: 0, fast processed: 0, slow path: 0
fast lock acquisition failures: 0, slow path: 0

Inderjeet Singh · ‎03-05-2014

Berry

you need to filter routes so only your LAN routes can get advertise to both peers.. It seems routes are getting rejected from ISP on second peer and vice versa by your router... check the logs as well if you see any specific message..

Thanks

Inder

bberry · ‎03-05-2014

Inderjeet,

Nothing in the log since we turned up the new link. Something I need to set to capture this?

Mar 3 13:14:07.725 EST: %BGP-5-ADJCHANGE: neighbor 10.1.4.58 Up

Mar 3 15:53:14.425 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 3 16:00:09.105 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 3 16:16:59.757 EST: %CLEAR-5-COUNTERS: Clear counter on interface GigabitEthernet0/2 by wberry on vty0 (172.16.6.9)

Mar 4 09:23:51.800 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 4 17:49:13.895 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 5 09:43:58.652 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 5 09:46:40.012 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 3 13:14:07.725 EST: %BGP-5-ADJCHANGE: neighbor 10.1.4.58 Up

Mar 3 15:53:14.425 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 3 16:00:09.105 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)
Mar 3 16:16:59.757 EST: %CLEAR-5-COUNTERS: Clear counter on interface GigabitEthernet0/2 by wberry on vty0 (172.16.6.9)

Mar 4 09:23:51.800 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 4 17:49:13.895 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 5 09:43:58.652 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Mar 5 09:46:40.012 EST: %SYS-5-CONFIG_I: Configured from console by wberry on vty0 (172.16.6.9)

Umesh Shetty · ‎03-05-2014

Hi,

It could possible be that the new link is receiving routes bu the best routes are still via the old link. Assuming

10.1.4.58 is the neigbor address of the new link can you please send the output of the below commands

show ip bgp summary

show ip bgp 10.1.2.49/32

a snippet of show ip bgp

a snippet of show ip bgp neigbor 10.1.4.58 received-routes

HTH

Regards

Umesh

bberry · ‎03-05-2014

Umesh,

Here you go ...

sh ip bgp summary
BGP router identifier 10.1.2.45, local AS number 65006
BGP table version is 267067, main routing table version 267067
290 network entries using 39440 bytes of memory
293 path entries using 16408 bytes of memory
42/41 BGP path/bestpath attribute entries using 5376 bytes of memory
30 BGP AS-PATH entries using 720 bytes of memory
4 BGP community entries using 112 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 62056 total bytes of memory
BGP activity 23955/23665 prefixes, 187204/186911 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer InQ OutQ Up/Down State/PfxRcd
10.1.4.46       4        13979   95244   98874   267067    0    0 8w5d          254
10.1.4.58       4        13979    2715    3045   267067    0    0 1d21h           2

sh ip bgp 10.1.2.49/32

BGP routing table entry for 10.1.2.49/32, version 265602

Paths: (1 available, best #1, table default)

Advertised to update-groups:

19

13979 65010, (received & used)

10.1.4.46 from 10.1.4.46 (12.123.67.172)

Origin IGP, localpref 100, valid, external, best

snippet of sh ip bgp
BGP table version is 267067, local router ID is 10.1.2.45
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.4.46                              0 13979 4473 i
*> 10.1.1.132/30    10.1.4.58                              0 13979 ?
*> 10.1.2.17/32     10.1.4.46                              0 13979 65002 ?
*> 10.1.2.25/32     10.1.4.46                              0 13979 65004 ?
*> 10.1.2.29/32     10.1.4.46                              0 13979 65026 ?
*> 10.1.2.33/32     10.1.4.46                              0 13979 65000 ?
*> 10.1.2.37/32     10.1.4.46                              0 13979 65000 ?
*> 10.1.2.41/32     10.1.4.46                              0 13979 65008 ?
*> 10.1.2.45/32     0.0.0.0                100         32768 ?
*> 10.1.2.49/32     10.1.4.46                              0 13979 65010 i
*> 10.1.2.53/32     10.1.4.46                              0 13979 65012 ?
*> 10.1.2.61/32     10.1.4.46                              0 13979 65033 ?
*> 10.1.2.73/32     10.1.4.46                              0 13979 65030 ?
*> 10.1.2.77/32     10.1.4.46                              0 13979 65017 ?
*> 10.1.2.81/32     10.1.4.46                              0 13979 65018 ?
*> 10.1.2.85/32     172.17.1.9             100         32768 ?

sh ip bgp neighbors 10.1.4.58 received-routes
BGP table version is 267067, local router ID is 10.1.2.45
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.1.132/30    10.1.4.58                              0 13979 ?
* 10.1.4.56/30     10.1.4.58                0             0 13979 ?

Total number of prefixes 2

Umesh Shetty · ‎03-05-2014

Hi

sh ip bgp 10.1.2.49/32

BGP routing table entry for 10.1.2.49/32, version 265602

Paths: (1 available, best #1, table default)

Advertised to update-groups:

19

13979 65010, (received & used)

10.1.4.46 from 10.1.4.46 (12.123.67.172)

Origin IGP, localpref 100, valid, external, best

Shows only 1 path for this prefix , when it ideally should be two.

sh ip bgp neighbors 10.1.4.58 received-routes

BGP table version is 267067, local router ID is 10.1.2.45
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.1.132/30    10.1.4.58                              0 13979 ?
* 10.1.4.56/30     10.1.4.58                0             0 13979 ?

Total number of prefixes 2

The above command shows the raw unprocessed routes received from the peer. If you have two peers advertising the same route the routing table will contain the best route or both incase they have the same metrics. In your case the above command shows only two prefixes received from this peer which means that the ISP is not advertising all the routes to you which is why the 3 Mb T1 link is being prefered which provides specific routes.

First thing you need to contact your ISP and figure out why aren't they sending the routes over this neighborship.

Secondly since you are using a 2921 which may not have the processing power to process BGP prefixes from two neighbors. What you can do is ask the ISP to send only a default route from the old link and send the more specific full routes over the new link. This will ensure all the traffic to internet will prefer the new 20 meg link instead of the old link.

Thirdly you need to ensure return traffic from the internet to your network must also prefer the 20 meg link. You can use the below config to prepend an additional AS in your route advertisements via the old link so that the AS path in the route updates via the New link is shorter than the old link and gets preference.

route-map ASPATH-PREPEND permit 10

set as-path prepend 65006

router bgp 65006

neighbor 10.1.4.46 route-map ASPATH-PREPEND out

Do let me know if this helps !

Please rate helpful posts

Regards

Umesh

Inderjeet Singh · ‎03-05-2014

Umesh,

I am agree with you that ISP is blocking routes and not advertising complete routes..

but I am not agree that 2921 is not capable of handling two BGP peers.. 29xx is ISR G2 series routers which are quite good in processing power and I am not sure they are able to handle muliple BGP peers..

thanks

Inder

Umesh Shetty · ‎03-05-2014

Inderjeet,

Agree they are ISR G2 and are better than the G1's but in a situation where you need only one BGP peer to be preffered why put additional load on the router to select the best path.

Regards

Umesh

JohnTylerPearce · ‎03-05-2014

Inder,

As far as the 2921 goes, that depends on how many routes you are taking in with the BGP peers. Typically you would want aroud 512MBs for a full Internet routing table. Umesh posted a great solution, but I do have one thing to add. Do you need a full Internet routing table, since you want traffic primarily to go out of the 20MEG Metro circuit, which obviously makes a lot of sense? You can always setup IP SLA or something similar to switch over traffic if the 20MEG Metro circuit goes down.

Just an idea..

bberry · ‎03-05-2014

I am thinking that the ISP is doing something and have a call setup to discuss with them. I figure there is something that needs to be tweaked since they were probably not expecting this to be in a router with another link.

In discussing with management when we ordered we internally discussed what we were going to do with the 3M since this is going to grow into a DR location eventually. This will include replication between two SANs. This location has another 3M terminated in a completely different router so am thinking of the following for this router.

1) mark the SAN replication traffic to ONLY flow over the 20M. SANs can stage data while outage is addressed.

2) use the 3M as a backup for remaining traffic in the event the 20M drops

I just need to get a handle on how traffic now needs to flow and what needs to happen if primary links drop. Big picture would be all traffic flow over 20M and if drop have critical stuff flow over existing 3M. Just not sure how to get there yet from here.

In regards to the power of the router, I do not think that will be an issue. This same platform is also running VoIP in addition to BGP at several other locations without any issues.

I will update as I find out more from teh ISP.

bberry · ‎03-06-2014

The issue is with my provider. They did not properly configure the circuit. It was placed into a VPN group on our MPLS network that we are in the process of disconnecting instead of where it is supposed to be. Hence it is not properly distributing all our corporate routes.

Thanks ...

Brent