BGP Peering flapping - hold time expired

[Filip Knezevic]

Hello all. We are having a weird issue trying to get iBGP up between two 7609 and 6509. 7609 already has around 10 active iBGP/eBGP peerings. They are all working fine. The problem emerged with the introduction of a newly installed 6509 and trying to get iBGP peering up. The problem is BGP is tearing down the session stating the hello timers are expired.

I googled before posting of course and found no usable answers.

To rule out:

1. Not an MTU issue.  Peers are directly connected with 9200 MTU. I can ping the loopback on another router with the packet size of more than 9000.

2. Not a congestion issue. The router has 2x10G Uplink and no active services yet.

3. Timers are same on both routers. 180 and 60 sec.

I'm peering with the loopbacks. OSPF and MPLS are enabled, and loopbacks are inserted into the routing table as /32.

Config on one router:

Rigel#sh run part router bgp 9125
Building configuration...

Current configuration : 702 bytes
!
Configuration of Partition - router bgp 9125
!
!
router bgp 9125
bgp router-id 178.xxx.175.254
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor 178.253.xxx.250 remote-as 9125
neighbor 178.253.xxx.250 description ATLAS_iBGP
neighbor 178.253.xxx.250 update-source Loopback0
!
address-family ipv4
redistribute connected
redistribute static
neighbor 178.253.xxx.250 activate
neighbor 178.253.xxx.250 send-community both
neighbor 178.253.xxx.250 route-map Orion-Peering-in in
no auto-summary
no synchronization
exit-address-family
!
address-family vpnv4
neighbor 178.253.xxx.250 activate
neighbor 178.253.xxx.250 send-community both
exit-address-family
!
!
end

Peering is constantly flapping.

May 25 11:28:32.556: %BGP-3-NOTIFICATION: received from neighbor 178.254.xxx.254 4/0 (hold time expired) 0 bytes
May 25 11:28:32.556: %BGP-5-ADJCHANGE: neighbor 178.254.xxxx.254 Down BGP Notification received
May 25 11:28:32.616: %BGP_SESSION-5-ADJCHANGE: neighbor 178.254.xxx.254 VPNv4 Unicast topology base removed from session BGP Notification received
May 25 11:28:32.616: %BGP_SESSION-5-ADJCHANGE: neighbor 178.254.xxx254 IPv4 Unicast topology base removed from session BGP Notification received
May 25 11:28:44.761: %BGP-5-ADJCHANGE: neighbor 178.254.xxx.254 Up

Here are some of debugs:

TPA: Reserved port 0 in Transport Port Agent for TCP IP type 1
1w2d: TCP: connection attempt to port 8291
1w2d: TCP: sending RST, seq 0, ack 4043267601
1w2d: TCP: sent RST to 178.253.xxx.153:64983 from 178.253.xxx.151:8291
1w2d: TCP0: state was LISTEN -> CLOSED [0 -> UNKNOWN(0)]
1w2d: TPA: Released port 0 in Transport Port Agent for TCP IP type 1 delay 240000
1w2d: TCB 0x5274B49C destroyed
Rigel#
1w2d: TPA: Reserved port 0 in Transport Port Agent for TCP IP type 1
1w2d: TCP: connection attempt to port 2086
1w2d: TCP: sending RST, seq 0, ack 1098242079
1w2d: TCP: sent RST to 107.xxx.231.45:51423 from 178.254.xxx.254:2086
1w2d: TCP0: state was LISTEN -> CLOSED [0 -> UNKNOWN(0)]
1w2d: TPA: Released port 0 in Transport Port Agent for TCP IP type 1 delay 240000
1w2d: TCB 0x52BCA0FC destroyed
Rigel#
1w2d: BGP: Regular scanner event timer
1w2d: BGP(4): Import timer expired. Walking from 1 to 1
1w2d: BGP(5): Import timer expired. Walking from 1 to 1
Rigel#
1w2d: BGP: 178.253.xxx.250 connection timed out 180600ms (last update) 180000ms (hold time)
1w2d: BGP: 178.253.xxx.250 went from Established to Closing
1w2d: BGP: 178.253.xxx.250 reset due to BGP Notification sent
May 25 09:09:32.445: %BGP-5-ADJCHANGE: neighbor 178.253.xxx.250 Down BGP Notification sent
May 25 09:09:32.445: %BGP-3-NOTIFICATION: sent to neighbor 178.253.xxx.250 4/0 (hold time expired) 0 bytes

[Georg Pauwen]

Hello,

the first thing I would do is check for basic connectivity issues between both peers. Can you post the output of 'sh interfaces x' where 'x' are the outgoing interfaces of both sides ?

Also, configure a simple IP SLA to ping the other side and check the statistics:

ip sla 1
icmp-echo 178.253.xxx.254
timeout 300
frequency 30
!
ip sla schedule 1 life forever start-time now

sh ip sla statistics 1

[Filip Knezevic]

Hello Georg,

Just to note I can extended ping with 9000 MTU with no issues.

Rigel#ping 178.253.xxx.250 size 9000 repeat 100000

Success rate is 100 percent (32533/32533), round-trip min/avg/max = 1/3/136 ms
Rigel#

Atlas#ping 178.254.xxx.254 size 9000 repeat 100000

Success rate is 100 percent (29348/29348), round-trip min/avg/max = 1/3/140 ms
Atlas#

Sh int outputs:

Atlas#sh int Po7
Port-channel7 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 8843.e1df.0800 (bia 8843.e1df.0800)
  Description: RIGEL
  Internet address is 178.253.xxx.150/31
  MTU 9200 bytes, BW 40000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 255/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  Transport mode LAN (10GBASE-R, 10.3125Gb/s)
  input flow-control is off, output flow-control is off
  Members in this channel: Te3/3 Te3/4 Te8/1 Te8/2
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 4527000 bits/sec, 94 packets/sec
  30 second output rate 4516000 bits/sec, 76 packets/sec
  L2 Switched: ucast: 202757 pkt, 585338338 bytes - mcast: 98753 pkt, 27061092 bytes
  L3 in Switched: ucast: 201046 pkt, 25487553 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 117192 pkt, 7900601 bytes mcast: 0 pkt, 0 bytes
     530124 packets input, 597016043 bytes, 0 no buffer
     Received 138086 broadcasts (0 IP multicasts)
     0 runts, 60548 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     542984 packets output, 666143983 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Atlas#

Rigel#sh int Port-channel7
Port-channel7 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 00d0.0132.1000 (bia 00d0.0132.1000)
Description: Veza ka Atlasu
Internet address is 178.253.xxx.151/31
MTU 9200 bytes, BW 40000000 Kbit, DLY 10 usec,
reliability 249/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is unknown
input flow-control is on, output flow-control is off
Members in this channel: Te7/3 Te7/4 Te8/3 Te8/4
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 04:37:09
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 3621000 bits/sec, 58 packets/sec
30 second output rate 3631000 bits/sec, 72 packets/sec
L2 Switched: ucast: 83527 pkt, 559479910 bytes - mcast: 17314 pkt, 4463402 bytes
L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes
105557 packets input, 566046194 bytes, 0 no buffer
Received 20925 broadcasts (0 IP multicasts)
0 runts, 62843 giants, 0 throttles
1138 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
131437 packets output, 566338759 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Rigel#

I've already determined there are increasing input errors on the connected interface. I've cleared the counters maybe four hours ago, it already has 1138 input errors.

I have another iBGP session and it's working well, as you can see by the uptime.

sh ip bgp sum

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
178.253.xxx.250 4        9125     348    1183      222    0    0 00:01:21        0
178.253.xxx.253 4        9125     644     489      222    0    0 05:37:34      108

IP SLA output:

Rigel(config)#ip sla 1
Rigel(config-ip-sla)#icmp-echo 178.253.xxx.250
Rigel(config-ip-sla-echo)#timeout 300
Rigel(config-ip-sla-echo)#frequency 30
Rigel(config-ip-sla-echo)#ip sla schedule 1 life forever start-time now
Rigel(config)#end

Rigel#sh ip sla statistics 1
Round Trip Time (RTT) for       Index 1
Type of operation: icmp-echo
        Latest RTT: 1 ms
Latest operation start time: 20:06:00.491 CEST Fri May 25 2018
Latest operation return code: OK
Number of successes: 4
Number of failures: 0
Operation time to live: Forever


Rigel#sh ip sla statistics ?
  <1-2147483647>  Entry Number
  aggregated      IP SLAs Statistics Aggregated
  details         Detailed Output
  |               Output modifiers
  <cr>

Rigel#sh ip sla statistics 1 det
Rigel#sh ip sla statistics 1 details
Round Trip Time (RTT) for       Index 1
Type of operation: icmp-echo
        Latest RTT: 1 ms
Latest operation start time: 20:06:30.491 CEST Fri May 25 2018
Latest operation return code: OK
Over thresholds occurred: FALSE
Number of successes: 5
Number of failures: 0
Operation time to live: Forever
Operational state of entry: Active
Last time this entry was reset: Never


Rigel#

[Georg Pauwen · Accepted solution]

Hello,

Rigel#sh int Port-channel7
Port-channel7 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 00d0.0132.1000 (bia 00d0.0132.1000)
Description: Veza ka Atlasu
Internet address is 178.253.xxx.151/31
MTU 9200 bytes, BW 40000000 Kbit, DLY 10 usec,
reliability 249/255, txload 1/255, rxload 1/255

That is most likely your problem. Anything other than 255/255 points to possible physical problems. Check your cabling, and/or switch the channel members to other ports...

[Filip Knezevic]

Hello George,

Thanks for pointing that out! 

I will let you know if that worked.

[Harold Ritter]

The "hold time expired" is very often related to a different TCP maximum segment size (MSS) use by both neighbors. In you case You can't really exclude this possibility even if the two neighbors are directly connected, since you use the loopback interface address to establis the session.

Can you run the following command on both sides just to verify the MSS that BGP uses.

"sh bgp ipv4 uni neighbors <neighbor address> | incl Datagrams"

Regards,

[Filip Knezevic]

Guys, thanks for your answers. Will post needed outputs when I get home.

Please dont abandon his thread just yet. ☺

[Filip Knezevic]

Harold, here are the outputs you wanted to see:

Atlas#sh bgp ipv4 uni neighbors 178.254.175.254 | incl Datagrams
Datagrams (max data segment is 9136 bytes):

Rigel#sh bgp ipv4 uni neighbors 178.253.194.250 | incl Datagrams
Datagrams (max data segment is 9136 bytes):

Just to note that I did another iBGP session with a different core router and it's working properly.

[Harold Ritter]

You can definitely rule out MSS issues at this point.

Rehards,

[marquisb.thibodeau]

Hi,

It look like I have the same issue (BGP flapping)... and when I do the command... I noticed that the "Maximum Data Segment" is higher then the TCP adjust-mss value...

Could it be the problem?

THank you

Marquis

[Filip Knezevic]

Hi George,

Thank you for pointing to reliability of the interface. That led us to further investigate. The problem was not in Layer 1, but in the MTU mismatch. We had a system jumbomtu 9000 command, and the MTU on the interface of 9200. We changed it 9200 and now it's all good.

[mauricio.ayala.barahona]

Hi, I have a similar issue, my question is if the interfaces of the peerings have different MTU can be the problem?

Peer 1:

GigabitEthernet3/33 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet Port, address is XXXX.XXXX.XXXX

  Description: ""

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

Peer 2:

Interface state transitions: 17

  Hardware is GigabitEthernet, address is XXXX.XXXX.XXXX

  Description: ""

  Internet address is Unknown

  MTU 9216 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)

Thanks for your answer!