11-24-2022 10:45 PM - edited 11-24-2022 10:50 PM
Hi all,
We have two Core N7K running fine and on one of the C9407R we get these error logs few times a day, looks like osfp/bfd flapping but i cant detect the reason:
21:13:21.576: %BFDFSM-6-BFD_SESS_DOWN: BFD-SYSLOG: BFD session ld:24 handle:2,is going Down Reason: ECHO FAILURE
Nov 24 21:13:21.576: %BFD-6-BFD_SESS_DESTROYED: BFD-SYSLOG: bfd_session_destroyed, ld:24 neigh proc:OSPF, handle:2 act
Nov 24 21:13:21.576: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.578: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.600: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.638: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.646: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.647: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.647: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:23.371: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from LOADING to FULL, Loading Done
Nov 24 21:13:23.371: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 15.7.1.33 proc:OSPF, idb:TenGigabitEthernet1/3/0/1.432 handle:3 act
Nov 24 21:13:23.832: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from LOADING to FULL, Loading Done
Nov 24 21:13:24.012: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from LOADING to FULL, Loading Done
Nov 24 21:13:25.269: %SW_MATM-4-MACFLAP_NOTIF: Host f8b4.aaa.aaa in vlan 244 is flapping between port Po81 and port Po142
Nov 24 21:13:26.317: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:34 handle:3 is going UP
Nov 24 21:13:26.447: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from LOADING to FULL, Loading Done
Nov 24 21:13:26.801: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from LOADING to FULL, Loading Done
Nov 24 21:13:28.004: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from LOADING to FULL, Loading Done
Nov 24 21:13:28.165: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from LOADING to FULL, Loading Done
Nov 25 12:44:13.249: %BFDFSM-6-BFD_SESS_DOWN: BFD-SYSLOG: BFD session ld:24 handle:2,is going Down Reason: ECHO FAILURE
Nov 25 12:44:13.249: %BFD-6-BFD_SESS_DESTROYED: BFD-SYSLOG: bfd_session_destroyed, ld:24 neigh proc:OSPF, handle:2 act
Nov 25 12:44:13.249: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.261: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.264: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.1 on TenGigabitEthernet1/3/0/1.496 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.292: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.372: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.375: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.405: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.410: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.431: %OSPF-5-ADJCHG: Process 254, Nbr 10.254.0.2 on TenGigabitEthernet2/3/0/1.561 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:14.086: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from LOADING to FULL, Loading Done
Nov 25 12:44:14.086: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 15.4.1.49 proc:OSPF, idb:TenGigabitEthernet2/3/0/1.497 handle:7 act
Nov 25 12:44:14.501: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from LOADING to FULL, Loading Done
Nov 25 12:44:15.052: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from LOADING to FULL, Loading Done
Nov 25 12:44:15.414: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:24 handle:2 is going UP
Nov 25 12:44:16.351: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from LOADING to FULL, Loading Done
Nov 25 12:44:17.375: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from LOADING to FULL, Loading Done
Nov 25 12:44:17.768: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.1 on TenGigabitEthernet1/3/0/1.496 from LOADING to FULL, Loading Done
Nov 25 12:44:21.335: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from LOADING to FULL, Loading Done
Nov 25 12:44:21.335: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 15.7.1.33 proc:OSPF, idb:TenGigabitEthernet1/3/0/1.432 handle:3 act
Nov 25 12:44:21.743: %OSPF-5-ADJCHG: Process 254, Nbr 10.254.0.2 on TenGigabitEthernet2/3/0/1.561 from LOADING to FULL, Loading Done
Nov 25 12:44:22.639: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:34 handle:3 is going UP
Nov 25 12:44:22.703: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from LOADING to FULL, Loading Done
is this flapping issue? we checked cabling looks fine; CPU is max 30-50%;
Best Regards
11-24-2022 11:59 PM
can I see BFD config ?
11-25-2022 03:34 AM
let me know what part of it ? here example interface config:
interface Ethernet8/33.599
bandwidth 10000000
encapsulation dot1q 333
vrf member AA-BB
bfd interval 250 min_rx 250 multiplier 3
bfd echo-rx-interval 50
bfd ipv4 interval 50 min_rx 50 multiplier 3
bfd ipv6 interval 50 min_rx 50 multiplier 3
bfd ipv4 echo-rx-interval 50
bfd ipv6 echo-rx-interval 50
no ip redirects
ip address a.b.c.d/28
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 1 area
no shutdown
11-25-2022 03:56 AM
can you increase the BFD time into 300 300 3 and check again?
bfd ipv4 interval 50 min_rx 50 multiplier 3
11-27-2022 10:03 PM - edited 11-27-2022 10:05 PM
we will try it, do we need to schedule maintenance in advance as this is production switch ?
we see these errors twice a day and mainly on the above two interfaces while we have many others configured the same way ?
11-28-2022 12:42 AM - edited 11-28-2022 12:57 AM
so it not new config ? if yes ? let me check.
I notice this log message, can you draw topology and more info. about vlan 244 ??
%SW_MATM-4-MACFLAP_NOTIF: Host f8b4.aaa.aaa in vlan 244 is flapping between port Po81 and port Po142 Nov 24 21:13:26.317: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:34 handle:3 is going UP
11-28-2022 12:53 AM
Hello
Suggest to disable BFD on the interface and monitor the peering without fast detection,
interface Ethernet8/33.599
ip ospf bfd disable
if your peering is then stable re-enable BFD once again and then set the interface to just have the single tranmsit/receive interval.
interface Ethernet8/33.599
no bfd echo-rx-interval 50
no bfd ipv4 interval 50 min_rx 50 multiplier 3
no bfd ipv6 interval 50 min_rx 50 multiplier 3
no bfd ipv4 echo-rx-interval 50
no bfd ipv6 echo-rx-interval 50
bfd interval 50 min_rx 50 multiplier 3
ip ospf bfd enable
Note: The above changes should not effect the current ospf peering
11-28-2022 01:44 AM
Hello,
is your network actually experiencing outages (at the times indicated in your log files) ? Can you post the output of 'show bfd neighbors detail' ?
It might just be the 'bfd echo-rx-interval 50' being too low. You could try and set it to something higher (such as 250, or 500). Obviously, since this is a production environment, do this after hours.
12-12-2022 03:31 AM - edited 12-12-2022 03:33 AM
we have found that there were messages
TenGigabitEthernet1/3/0/1.404 from FULL to DOWN, Neighbor Down: BFD node down
ECHO FAILURE was the reason hence we entered:
no bfd echo - on all interfaces where we seen this and since then all works fine
12-12-2022 04:13 AM
I will run lab I have found some doc. about BFD in NSK.
if I found something I will update you.
12-13-2022 03:44 AM
SVI limitations:
An ASIC reset causes traffic disruption for other ports and it can cause the SVI sessions on the other ports to flap. For example, if the carrier interface is a virtual port channel (vPC), BFD is not supported over the SVI interface and it could cause a trigger for an ASIC reset. When a BFD session is over SVI using virtual port channel (vPC) Peer-Link, the BFD echo function is not supported. You must disable the BFD echo function for all sessions over SVI between vPC peer nodes.
An SVI on the Cisco Nexus series switches should not be configured to establish a BFD neighbor adjacency with a device connected to it via a vPC. This is because the BFD keepalives from the neighbour, if sent over the vPC member link connected to the vPC peer-switch, do not reach this SVI causing the BFD adjacency to fail.
When you change the topology (for example, add or delete a link into a VLAN, delete a member from a Layer 2 port channel, and so on), the SVI session could be affected. It may go down first and then come up after the topology discovery is finished.
BFD over FEX HIF interfaces is not supported.
When a BFD session is over SVI using virtual port-channel (vPC) Peer-Link (either BCM or GEM based ports), the BFD echo function is not supported. You must disable the BFD echo function for all sessions over SVI between vPC peer nodes using the no bfd echo command at the SVI configuration level.
according to Cisco Doc. BFD have limitation in SVI, are you config BFD over SVI ??
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide