Solved: Bgp Neighbour reset excatly every 3 minutes

kumarpmt83 · ‎05-19-2012

Hi,

i have cisco 1841 router and we have MPLS WAN Co.nnectivity

We configured BGP in the router and the link is not stable.

i checked in the router after every 3 minutes bgp session is resetted.

Kindly anyone tell me how to solve the problem.

Please find the below log for your reference.

LOG:

May 19 13:53:37.612 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 13:56:38.224 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 13:56:38.224 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 13:56:41.180 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 13:59:41.520 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 13:59:41.520 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 13:59:56.932 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:02:57.112 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:02:57.112 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:02:59.492 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:05:59.496 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:05:59.496 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:06:11.092 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:09:12.008 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:09:12.008 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:09:23.668 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:12:24.532 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:12:24.532 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:12:28.456 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:15:28.860 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:15:28.860 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:15:31.656 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:18:32.160 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:18:32.160 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:18:44.300 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:21:44.672 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:21:44.672 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:22:01.064 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:25:01.288 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:25:01.288 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:25:04.512 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:28:06.644 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:28:06.644 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:28:09.160 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:31:05.128 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down Interface

flap

May 19 14:31:07.116 India: %LINK-5-CHANGED: Interface FastEthernet0/1, changed s

tate to administratively down

May 19 14:31:08.116 India: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastE

thernet0/1, changed state to down

May 19 14:31:18.788 India: %SYS-5-CONFIG_I: Configured from console by vty0 (10.

10.43.241)

May 19 14:31:52.920 India: %SYS-5-CONFIG_I: Configured from console by vty0 (10.

10.43.241)

May 19 14:31:53.304 India: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed st

ate to up

May 19 14:31:54.304 India: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastE

thernet0/1, changed state to up

May 19 14:32:20.236 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:35:20.828 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:35:20.828 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:35:23.852 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:36:05.388 India: %SYS-5-CONFIG_I: Configured from console by vty0 (10.

10.43.241)

May 19 14:38:24.136 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:38:24.136 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:38:36.560 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:38:47.672 India: %SYS-5-CONFIG_I: Configured from console by vty0 (10.

10.43.241)

May 19 14:39:55.088 India: %SYS-5-CONFIG_I: Configured from console by vty0 (10.

10.43.241)

May 19 14:41:36.652 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:41:36.652 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:41:40.104 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:44:40.208 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:44:40.208 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:44:55.980 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

May 19 14:47:56.816 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Down BGP Notif

ication sent

May 19 14:47:56.816 India: %BGP-3-NOTIFICATION: sent to neighbor 10.237.48.5 4/0

(hold time expired) 0 bytes

May 19 14:48:01.876 India: %BGP-5-ADJCHANGE: neighbor 10.237.48.5 Up

TN-Trichy-Factory#

CSCO11190621 · ‎10-29-2013

Hi Guys, i faced a similar issue. In my case the MTU on the PE sub-interface was set to 2018 while the CE was at default 1500, changing it on the CE to 2018 fixed the issue. I don't know if I perhaps took a shortcut here, maybe there is a deeper underlying error. Please advise!

View solution in original post

Nagendra Kumar Nainar · ‎05-19-2012

Hi Dhinesh,

TN-Trichy-Factory router is not receiving any keepalive or BGP update from 10.237.48.5 for 3 minutes (hold timer) and so resetting the session. This normally happens when there is any issue with BGP Path MTU discovery.

Normally when PMTUD id enabled, any router will pull the IP MTU from egress interface, reduce 40 bytes (20 bytes for IP and 20 for TCP) and send the same as MSS in TCP SYN packet. once it negotiate the MSS in TCP handshake procedure, it will send the UPDATE upto MSS value and DF bit set in IP header.When any router along the path is not capable of sending the packet out due to lower MTU, will drop the packet and send ICMP error message with MTU value it can handle.

BGP originator iwll react to this message by sending the update again with size upto received value (again 40 bytes will be reduced on the received value). Any BGP router while sending an UPDATE will not send a keepalive (as this update can act as keepalive).

If for some reason, the intermittent router is not able to send the ICMP error message (due to drop @ Layer2 or blocked ICMP erros message), remote neighbor will neither receive update nor keepalive which will result in hold timer expiry and flapping the session. This condition will result in situation where the BGP will come up an dflap after 3 minutes (or every hold time) continously.

Check the MSS value used to send the update by 10.237.48.5 to your router. You can check the same using "show ip bgp neighbor <>" and check "max data segment ".

You can either disable path MTU discovery for this neighbor using "neighbor <> transport path-mtu-discovery disable" or you need to fix the MTU issue to get the session up and active.

HTH,

Nagendra

kumarpmt83 · ‎05-20-2012

As per your instruction i diasble the transport path-mtu-discovery again i reset every 3 minutes.

how to resolve this issue.

pls help me.

Nagendra Kumar Nainar · ‎05-20-2012

Hi Dhinesh,

What is the MSS value on both side?. You need to check the MSS on 10.237.48.5. for this neighbor. See if it is 536 or some other value.

try disabling on 10.237.48.5 as well.

-Nagendra

kumarpmt83 · ‎05-20-2012

I checked in my router and give command as per your instruction see the output below:

Any problem in the below output.

TN-Trichy-Factory#sh ip bgp neighbors 10.237.48.5 | i max

Number of NLRIs in the update sent: max 149, min 0

TN-Trichy-Factory#sh ip bgp ne

TN-Trichy-Factory#sh ip bgp neighbors 10.237.48.5

BGP neighbor is 10.237.48.5, remote AS 65500, external link

Description: ***TULIP WAN***

BGP version 4, remote router ID 124.247.252.83

BGP state = Established, up for 00:00:21

Last read 00:00:21, last write 00:00:19, hold time is 180, keepalive interval

is 60 seconds

Neighbor capabilities:

Route refresh: advertised and received(new)

New ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 475 475

Notifications: 473 0

Updates: 3464 150

Keepalives: 1775 496

Route Refresh: 0 0

Total: 6120 1121

Default minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast

BGP table version 4342, neighbor version 4342/0

Output queue size : 0

Index 2, Offset 0, Mask 0x4

2 update-group member

Inbound soft reconfiguration allowed

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 304 0

Prefixes Total: 304 0

Implicit Withdraw: 0 0

Explicit Withdraw: 0 0

Used as bestpath: n/a 0

Used as multipath: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

Total: 0 0

Number of NLRIs in the update sent: max 149, min 0

Address tracking is enabled, the RIB does have a route to 10.237.48.5

Connections established 475; dropped 474

Last reset 00:00:40, due to BGP Notification sent, hold time expired

Transport(tcp) path-mtu-discovery is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 10.237.48.6, Local port: 179

Foreign host: 10.237.48.5, Foreign port: 13724

Connection tableid (VRF): 0

Enqueued packets for retransmit: 1, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x5B93710):

Timer Starts Wakeups Next

Retrans 8 4 0x5B9571C

TimeWait 0 0 0x0

AckHold 2 1 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 0 0 0x0

DeadWait 0 0 0x0

Linger 0 0 0x0

ProcessQ 0 0 0x0

iss: 323124200 snduna: 323125733 sndnxt: 323125979 sndwnd: 16384

irs: 3209385506 rcvnxt: 3209385585 rcvwnd: 16306 delrcvwnd: 78

SRTT: 99 ms, RTTO: 1539 ms, RTV: 1440 ms, KRTT: 24624 ms

minRTT: 28 ms, maxRTT: 300 ms, ACK hold: 200 ms

Status Flags: passive open, gen tcbs

Option Flags: nagle

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 5 (out of order: 0), with data: 2, total data bytes: 78

Sent: 6 (retransmit: 4, fastretransmit: 0, partialack: 0, Second Congestion: 0),

with data: 4, total data bytes: 1778

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

Kishore Chennupati · ‎05-20-2012

Dinesh,

As narendra mentioned when yout type the below command you should see something like below Check the MSS in your case. Also you can do some pings with df-bit set to see what is the maximum packet you could send across and then check the path to make sure that the MTU is uniform or allowed to accept this BGP update packet

rt#sh ip bgp neighbors x.x.x.x | i max

Number of NLRIs in the update sent: max 3, min 0

minRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 ms

Datagrams (max data segment is 1460 bytes):

HTH

Kishore

kumarpmt83 · ‎05-20-2012

I checked in my router and give command as per your instruction see the output below:

Any problem in the below output.

TN-Trichy-Factory#sh ip bgp neighbors 10.237.48.5 | i max

Number of NLRIs in the update sent: max 149, min 0

TN-Trichy-Factory#sh ip bgp ne

TN-Trichy-Factory#sh ip bgp neighbors 10.237.48.5

BGP neighbor is 10.237.48.5, remote AS 65500, external link

Description: ***TULIP WAN***

BGP version 4, remote router ID 124.247.252.83

BGP state = Established, up for 00:00:21

Last read 00:00:21, last write 00:00:19, hold time is 180, keepalive interval

is 60 seconds

Neighbor capabilities:

Route refresh: advertised and received(new)

New ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 475 475

Notifications: 473 0

Updates: 3464 150

Keepalives: 1775 496

Route Refresh: 0 0

Total: 6120 1121

Default minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast

BGP table version 4342, neighbor version 4342/0

Output queue size : 0

Index 2, Offset 0, Mask 0x4

2 update-group member

Inbound soft reconfiguration allowed

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 304 0

Prefixes Total: 304 0

Implicit Withdraw: 0 0

Explicit Withdraw: 0 0

Used as bestpath: n/a 0

Used as multipath: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

Total: 0 0

Number of NLRIs in the update sent: max 149, min 0

Address tracking is enabled, the RIB does have a route to 10.237.48.5

Connections established 475; dropped 474

Last reset 00:00:40, due to BGP Notification sent, hold time expired

Transport(tcp) path-mtu-discovery is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 10.237.48.6, Local port: 179

Foreign host: 10.237.48.5, Foreign port: 13724

Connection tableid (VRF): 0

Enqueued packets for retransmit: 1, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x5B93710):

Timer Starts Wakeups Next

Retrans 8 4 0x5B9571C

TimeWait 0 0 0x0

AckHold 2 1 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 0 0 0x0

DeadWait 0 0 0x0

Linger 0 0 0x0

ProcessQ 0 0 0x0

iss: 323124200 snduna: 323125733 sndnxt: 323125979 sndwnd: 16384

irs: 3209385506 rcvnxt: 3209385585 rcvwnd: 16306 delrcvwnd: 78

SRTT: 99 ms, RTTO: 1539 ms, RTV: 1440 ms, KRTT: 24624 ms

minRTT: 28 ms, maxRTT: 300 ms, ACK hold: 200 ms

Status Flags: passive open, gen tcbs

Option Flags: nagle

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 5 (out of order: 0), with data: 2, total data bytes: 78

Sent: 6 (retransmit: 4, fastretransmit: 0, partialack: 0, Second Congestion: 0),

with data: 4, total data bytes: 1778

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

komondal · ‎12-23-2012

I have faced the similar issue but later found that ebgp neighbor is flapping in every 3 mins due to wrong subnet of directly conected interfaces.

CSCO11190621 · ‎10-29-2013

Hi Guys, i faced a similar issue. In my case the MTU on the PE sub-interface was set to 2018 while the CE was at default 1500, changing it on the CE to 2018 fixed the issue. I don't know if I perhaps took a shortcut here, maybe there is a deeper underlying error. Please advise!

prashsin1 · ‎03-28-2014

Nicely explained Nagendra! one vote from me

I tried MSS and disabling transport path mtu discovery but no luck,

finally fixed by adjusting MTU size on the interface.