Re: BGP Peer Flapping

jacobyoby · ‎09-04-2018

I've trying to investigate a BGP flapping that showing different neighbors going down at different times.

It seems like there's something causing the link to go down but I cannot make any sense of the logs or what they are trying to tell me. I'm going to post the logs from today where the BGP peers flapped several times.

If anyone can point me in the right direction or give me some insight as to what's causing these flaps I'd greatly appreciate it.

Core-SW1

*Sep  4 08:42:56.282: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:43:16.810: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:43:47.550: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:44:18.298: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:44:52.114: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:45:10.170: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:47:51.118: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:48:07.302: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:48:08.846: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:48:24.134: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:48:40.090: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:49:39.934: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:49:58.370: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:50:10.198: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:51:05.686: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:51:25.966: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:52:19.406: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:52:32.518: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:53:22.886: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:53:37.022: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:54:37.014: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:55:23.802: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:55:38.862: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:55:40.406: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:56:02.038: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:56:34.818: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 08:57:35.254: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 08:58:07.014: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:58:40.838: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:58:57.258: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:59:28.018: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 08:59:48.694: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 09:01:48.522: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 09:02:17.210: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 09:02:48.962: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 09:03:15.586: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes
*Sep  4 09:03:49.390: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 12:44:10.525: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 12:57:19.921: %BGP-5-ADJCHANGE: neighbor 192.168.255.21 Down Peer closed the session
*Sep  4 12:57:24.553: %TRACKING-5-STATE: 5 interface Gi2/23 line-protocol Up->Down
*Sep  4 12:57:26.053: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.21 on GigabitEthernet2/23 from FULL to DOWN, Neighbor Down: Interface down or detached
*Sep  4 12:59:05.557: %TRACKING-5-STATE: 5 interface Gi2/23 line-protocol Down->Up
*Sep  4 12:59:35.557: %TRACKING-5-STATE: 5 interface Gi2/23 line-protocol Up->Down
*Sep  4 12:59:36.557: %TRACKING-5-STATE: 5 interface Gi2/23 line-protocol Down->Up
*Sep  4 13:00:24.085: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.21 on GigabitEthernet2/23 from LOADING to FULL, Loading Done
*Sep  4 13:00:56.641: %BGP-5-ADJCHANGE: neighbor 192.168.255.21 Up
*Sep  4 13:01:08.069: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up
*Sep  4 16:43:39.912: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Down Peer closed the session
*Sep  4 17:00:29.656: %BGP-5-ADJCHANGE: neighbor 198.202.20.145 Up

Core-RTR1

*Sep  4 21:40:31.607: %HWIC_1GE_SFP-6-STARTUP: Gigabit Ethernet initialized
*Sep  4 21:40:32.851: %ESWMRVL_FLTMG-5-NOTICE: Notice: FPGA Rev 0x27

*Sep  4 21:40:34.099: %ESWMRVL_FLTMG-5-NOTICE: Notice: FPGA Rev 0x27

*Sep  4 21:40:36.673: %MAINBOARD_SFP-6-INSERTED: Transceiver module 1000BaseSX SFP is inserted in GigabitEthernet0/1 port.
*Sep  4 21:40:36.723: %MAINBOARD_SFP-6-INSERTED: Transceiver module 1000BaseSX SFP is inserted in GigabitEthernet0/0 port.
*Sep  4 21:40:36.779: %LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to reset
*Sep  4 21:40:36.779: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to reset
*Sep  4 21:40:36.779: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to up
*Sep  4 21:40:36.779: %LINK-3-UPDOWN: Interface GigabitEthernet0/3, changed state to down
*Sep  4 21:40:36.779: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/0, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/2, changed state to up
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/3, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/0, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1/0, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1/1, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2/0, changed state to down
*Sep  4 21:40:39.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2/1, changed state to down
*Sep  4 21:40:42.101: %SYS-6-CLOCKUPDATE: System clock has been updated from 21:40:42 UTC Tue Sep 4 2018 to 13:40:42 PST Tue Sep 4 2018, configured from console by console.
*Sep  4 21:40:42.101: %SYS-6-CLOCKUPDATE: System clock has been updated from 13:40:42 PST Tue Sep 4 2018 to 14:40:42 PST Tue Sep 4 2018, configured from console by console.
*Sep  4 21:40:42.635: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.1.10.113 port 514 CLI Request Triggered
*Sep  4 21:40:42.635: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.1.10.178 port 514 CLI Request Triggered
*Sep  4 21:40:43.163: %SYS-5-CONFIG_I: Configured from memory by console
*Sep  4 21:40:43.469: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
*Sep  4 21:40:44.541: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to down
*Sep  4 21:40:44.775: %LINK-5-CHANGED: Interface GigabitEthernet0/3, changed state to administratively down
*Sep  4 21:40:44.775: %LINK-5-CHANGED: Interface GigabitEthernet0/0/0, changed state to administratively down
*Sep  4 21:40:44.775: %LINK-5-CHANGED: Interface FastEthernet0/1/0, changed state to administratively down
*Sep  4 21:40:44.775: %LINK-5-CHANGED: Interface FastEthernet0/1/1, changed state to administratively down
*Sep  4 21:40:44.775: %LINK-5-CHANGED: Interface FastEthernet0/2/0, changed state to administratively down
*Sep  4 21:40:44.775: %LINK-5-CHANGED: Interface FastEthernet0/2/1, changed state to administratively down
*Sep  4 21:40:47.721: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to up
*Sep  4 21:40:48.775: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Sep  4 21:40:49.001: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*Sep  4 21:40:49.809: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
*Sep  4 21:40:50.009: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*Sep  4 21:40:51.407: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C3900e Software (C3900e-UNIVERSALK9-M), Version 15.2(4)M6a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Tue 15-Apr-14 06:10 by prod_rel_team
*Sep  4 21:40:51.605: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Sep  4 21:40:51.695: %SNMP-5-COLDSTART: SNMP agent on host caph-rtr2 is undergoing a cold start
*Sep  4 21:40:53.139: %SYS-6-BOOTTIME: Time taken to reboot after reload =  152 seconds
*Sep  4 21:40:54.767: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.1.10.113 port 514 started - CLI initiated
*Sep  4 21:40:54.767: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.1.10.178 port 514 started - CLI initiated
*Sep  4 21:41:19.939: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.24 on GigabitEthernet0/2.1000 from LOADING to FULL, Loading Done
*Sep  4 21:41:24.517: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.22 on GigabitEthernet0/2.1000 from LOADING to FULL, Loading Done
*Sep  4 21:41:27.165: %BGP-5-ADJCHANGE: neighbor 67.131.8.149 Up
*Sep  4 21:41:30.119: %BGP-5-ADJCHANGE: neighbor 10.200.1.11 Up
*Sep  4 21:41:36.053: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.104 on GigabitEthernet0/1 from LOADING to FULL, Loading Done
*Sep  4 21:41:36.139: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.103 on GigabitEthernet0/0 from LOADING to FULL, Loading Done
*Sep  4 21:42:08.691: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
*Sep  4 21:42:14.673: %BGP-5-ADJCHANGE: neighbor 192.168.255.104 Up
*Sep  5 01:54:40.224: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (10.15.30.222)
Sep  5 04:32:15.783: %SYS-5-SCHEDULED_RELOAD: Reload requested for 21:42:12 PST Tue Sep 4 2018 at 21:32:12 PST Tue Sep 4 2018 by admin on vty0 (10.15.30.222). Reload Reason: Reload Command.
.Sep  5 04:33:06.549: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (10.15.30.222)
Sep  5 04:37:45.486: %SYS-5-SCHEDULED_RELOAD_CANCELLED:  Scheduled reload cancelled at 21:37:45 PST Tue Sep 4 2018

Edge-RTR1

078670: Sep  4 10:48:15.168 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078671: Sep  4 10:48:45.928 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078672: Sep  4 10:49:16.679 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078673: Sep  4 10:49:50.499 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078674: Sep  4 10:49:55.243 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down BGP Notification sent
078675: Sep  4 10:49:55.243 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 4/0 (hold time expired) 0 bytes
078676: Sep  4 10:49:55.243 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  BGP Notification sent
078677: Sep  4 10:50:08.559 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078679: Sep  4 10:52:52.397 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078680: Sep  4 10:52:52.397 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078681: Sep  4 10:53:05.704 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078682: Sep  4 10:53:22.540 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078683: Sep  4 10:53:22.540 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078684: Sep  4 10:53:22.540 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078685: Sep  4 10:53:38.496 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078686: Sep  4 10:54:56.783 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078687: Sep  4 10:54:56.787 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078688: Sep  4 10:54:56.787 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078689: Sep  4 10:55:08.615 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078691: Sep  4 10:56:13.526 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078692: Sep  4 10:56:13.526 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078693: Sep  4 10:56:24.386 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078694: Sep  4 10:57:20.842 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078695: Sep  4 10:57:20.842 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078696: Sep  4 10:57:30.945 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078697: Sep  4 10:58:24.329 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078698: Sep  4 10:58:24.329 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078699: Sep  4 10:58:35.461 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078700: Sep  4 11:00:22.252 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078701: Sep  4 11:00:22.256 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078702: Sep  4 11:00:22.256 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078703: Sep  4 11:00:37.312 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078704: Sep  4 11:01:00.492 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078705: Sep  4 11:01:22.932 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down Peer closed the session
078706: Sep  4 11:01:22.932 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  Peer closed the session
078707: Sep  4 11:01:33.280 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078710: Sep  4 11:03:05.484 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078711: Sep  4 11:03:39.311 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078712: Sep  4 11:03:55.723 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078713: Sep  4 11:04:26.487 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078714: Sep  4 11:04:33.859 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down BGP Notification sent
078715: Sep  4 11:04:33.859 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 4/0 (hold time expired) 0 bytes
078716: Sep  4 11:04:33.859 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  BGP Notification sent
078717: Sep  4 11:04:47.171 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078721: Sep  4 11:07:15.707 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078722: Sep  4 11:07:47.467 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078723: Sep  4 11:08:14.091 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 passive 6/0 (CEASE: unknown subcode) 0 bytes
078724: Sep  4 11:08:47.811 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down BGP Notification sent
078725: Sep  4 11:08:47.811 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 4/0 (hold time expired) 0 bytes
078726: Sep  4 11:08:47.811 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  BGP Notification sent
078727: Sep  4 11:08:47.903 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078764: Sep  4 14:51:21.469 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down BGP Notification sent
078765: Sep  4 14:51:21.469 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 4/0 (hold time expired) 0 bytes
078766: Sep  4 14:51:21.469 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  BGP Notification sent
078771: Sep  4 15:06:08.248 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up
078810: Sep  4 18:50:41.652 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Down BGP Notification sent
078811: Sep  4 18:50:41.652 PDT: %BGP-3-NOTIFICATION: sent to neighbor 192.168.255.103 4/0 (hold time expired) 0 bytes
078812: Sep  4 18:50:41.652 PDT: %BGP_SESSION-5-ADJCHANGE: neighbor 192.168.255.103 IPv4 Unicast topology base removed from session  BGP Notification sent
078815: Sep  4 19:05:31.530 PDT: %BGP-5-ADJCHANGE: neighbor 192.168.255.103 Up

REMOTESITE1-RTR1

*Sep  4 12:12:05.823: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.17 on GigabitEthernet0/2.1000 from FULL to DOWN, Neighbor Down: Dead timer expired
*Sep  4 12:15:13.721: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.17 on GigabitEthernet0/2.1000 from LOADING to FULL, Loading Done
*Sep  4 18:12:55.827: %BGP-5-ADJCHANGE: neighbor 10.200.1.1 Down Peer closed the session
*Sep  4 18:12:55.827: %BGP_SESSION-5-ADJCHANGE: neighbor 10.200.1.1 IPv4 Unicast topology base removed from session  Peer closed the session
*Sep  4 18:13:35.827: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.21 on GigabitEthernet0/2.1000 from FULL to DOWN, Neighbor Down: Dead timer expired
*Sep  4 18:15:48.339: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.255.21 on GigabitEthernet0/2.1000 from LOADING to FULL, Loading Done
*Sep  4 18:15:53.941: %BGP-5-ADJCHANGE: neighbor 10.200.1.1 Up
*Sep  5 02:17:57.583: %SYS-5-CONFIG_I: Configured from console by admin on vty1 (10.15.30.222)

Peter Paluch · ‎09-05-2018

Hi Jacob,

It might be a red herring but one particular repetitive message in your logs got my attention:

*Sep  4 08:43:16.810: %BGP-3-NOTIFICATION: received from neighbor 198.202.20.145 6/0 (cease) 0 bytes

The CEASE BGP notification message would refer to a connection collision in BGP. To explain, it is possible for two BGP neighbors to establish a TCP session to each other at the same time, resulting into two BGP sessions between a single pair of routers. This is because each BGP speaker decides on its own when to start opening a TCP session to its neighbor, and it is possible for the two BGP speakers to decide to open a session to each other around the same timeframe. In BGP parlance, this is called a connection collision. If this happens, the additional TCP sessions will be torn down so that in the end, only a single BGP session between the two neighbors stays up. The CEASE notification is exactly the message sent to a neighbor when tearing down a BGP session due to connection collision.

Now, seeing that message appear once over a long time is possible even though unlikely. However, seeing it repeated so often suggests that something is wrong.

I have seen a similar trace in situation when BGP sessions have been configured to be established using the virtual IP addresses provided by HSRP, VRRP, or GLBP (or any other FHRP protocol); the messages would start popping up if the Active (or Master) router role moved temporarily to a different router and then came back. I have to emphasize that using virtual IP addresses for BGP session endpoints would be a grave misconfiguration; this cannot and will not work properly.

Can you please verify whether your BGP configuration, at any point, refers to an HSRP/VRRP/GLBP vIP?

Best regards,
Peter

jacobyoby · ‎09-05-2018

I don't see any of those things but I'm not super familiar with complex BGP/OSPF deployments.

Would show run | sec bgp give me that info?

Peter Paluch · ‎09-05-2018

Hello Jacob,

Would show run | sec bgp give me that info?

Unfortunately not - or only very indirectly. You would need to look into the neighbor commands, take the IP addresses referenced in those commands, and verify their owning routers whether these IP addresses are simple addresses on interfaces, or whether they are configured as virtual IP addresses in first hop redundancy protocols.

So if we focused on the IP addresses of BGP speakers seen in the logs, how are the following IP addresses configured on their owning routers? Would the be configured on interfaces using the ip address command, or would the appear in standby/vrrp/glbp commands?

198.202.20.145
67.131.8.149
10.200.1.1
10.200.1.11
192.168.255.103
192.168.255.104

Thank you!

Best regards,
Peter