Strange BGP issue

[CliveG]

I have two data centres and one of them connects upstream and receives the full internet routing table this is then forwarded via iBGP to the other Data-Centre (Don't worry about if this is good practice or not, it is configuration I have inherited and can do nothing about for now).

With no change of configurations and no network changes, suddenly the holdown timers are expring and this connection is constantly up/down because of the peer resets.

Weirdly, we are able to ping devices connected to this DC but cannot pass any other traffic. Here is what I am now seeing in the error log and was hoping someone could point me in the right direction:

Mar 23 12:31:46.216: %LDP-SW1-5-SP: 192.168.1.1:0: session hold up initiated
Mar 23 12:32:42.903: %MSDP-SW1-5-PEER_UPDOWN: Session to peer 192.168.1.1 going down
Mar 23 12:33:03.944: %LDP-SW1-5-SP: 192.168.1.1 :0: session recovery succeeded
Mar 23 12:33:28.085: %MSDP-SW1-5-PEER_UPDOWN: Session to peer 192.168.1.1 going up
Mar 23 12:39:44.601: %LDP-SW1-5-SP: 192.168.1.1:0: session hold up initiated
Mar 23 12:40:31.532: %MSDP-SW1-5-PEER_UPDOWN: Session to peer 192.168.1.1 going down
Mar 23 12:40:55.721: %LDP-SW1-5-SP: 192.168.1.1:0: session recovery succeeded
Mar 23 12:41:28.127: %MSDP-SW1-5-PEER_UPDOWN: Session to peer 192.168.1.1 going up

I have confirmed the X-Connect is good and have also replaced the sfp's. I am planning on changing out the core switches as it could be hardware or an ios issue, but I am hoping I do not have to.

Thanks

[Harold Ritter]

@CliveG , If the service provider delivers a circuit with a given MTU, you should be able to ping the MTU value without fragmenting. You need to work with the SP to sort it out.

Regards,

[CliveG]

Hi Harold

Service provider states: Link-level type: Ethernet-Bridge, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None,

[MHM Cisco World]

can I see the ping ?

[CliveG]

When I ping from a VM located in DC-3 (see the diagram I attached in the thread) to another Baremetal system located in DC-1 (again, please see the diagram) with no extra switches to he ping (in other words, a normal ping) I get a response. But when I try and ping from the same device to the same device with the -l (size) and -f (do not fragment) bits set and the size is above the default, I get the following response:

Packet needs to be fragmented but DF set

But that is more than likely because the packet size 7976 is for the BGP sessions only (at a guess).

[MHM Cisco World]

Xconnect and ibgp?

Are you run l2vpn or l3vpn?

One more thing check the cpu see if the bgp is top 5 process in cpu sort 

[CliveG]

Hi MHM

No L2VPN or L3VPN.

VPLS, ISIS, iBGP, eBGP, MPLS, Multicast.

BGP is not in the list at all under the CPU Processes.

Basically, as we have discussed previously, we actually have 3 x DC's in a triangle. One DC is at base and the other two are in Telehouse in London. The two in Telehouse are connected via a X-Connect (10gbps Fibre). I have already got confirmation from Telehouse that the fibre is good. The Core systems at one location in telehouse has the upstream connectivity and receives the internet routing table via eBGP. This has been configured to advertise these tables, via eBGP to the other 2 DC's. We started seeing these problems a few days ago and I have so far completed the following:

reset iBGP peering between the 2 x telehouse DC's.

Confirmed routing between the two (ISIS - BGP)

Created a static route to over-ride the ISIS to the RID of the telehouse upstream DC.

Swapped sfp's.

Completed debugs and reloaded the VSS

The only thing that is left is a possible IOS Corruption or a hardware fault somewhere on the switch.

I want to try and eliminate all other possibilities before swapping out the switches with LAB ones that are coinfigured the same and work perfectly. This is where I am hoping you guys with tons of Ciso knowledge can help.

Many thanks for the ongoing help you guys are giving. It is very much appreciated.

[MHM Cisco World]

A BGP reset can be caused by any of the following events:

• A loss of connectivity between BGP peers <<- HERE the issue 
• Making a configuration change to the BGP parameters on one of the peers <<- you confirm no change done 
• Manually clearing BGP sessions by an administrator <<- this you run only after do some testing so no problem 
  
  

NOW the connectivity is check by hello (keepalive message)
here you must check 
A-
router generate the hello message 
 R2#show ip bgp neighbors 1.1.1.1
Last read 00:00:15, last write 00:00:44, hold time is 180, <<- last write is last time this peer generate hello message 
keepalive interval is 60 second
B- check if the router can send the hello to neighbor 
R2#show ip bgp sum | begin Neighbor
Neighbor … MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
                                  X
then wait 
R2#show ip bgp sum | begin Neighbor
Neighbor … MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
                                  X+1 or more

[CliveG]

Yes, the holdown timers expire. The session is resetting every 3 minutes (180 seconds). But I can ping the RID of the peer. I can also ping devices attached to either of these peers and can also traceroute to them. What seems to be happening is ICMP is getting through and nothing else is.

[CliveG]

If it helps to attach Configurations of all 3 systems I can do that. Will take a while as I will need to hide all the public addresses we utilise.

[paul driver]

Hello
Can you post the output of the following please?

sh tcp brief numeric
sh tcp tcb x.x.x.x.
sh ip traffic | in echo
sh ip bgp ipv4 unicast neighbors x.x.x.x | in TTL
sh ip bgp ipv4 unicast neighbors x.x.x.x | sec Last reset

[CliveG]

Hi Paul,

Sure,

System 1:

show tcp brief numeric - command "numeric" does not exist on the system

show tcp tcb 18138098
Connection state is ESTAB, I/O status: 1, unread input bytes: 0 - says ESTAB but is actually in idle
Mininum incoming TTL 0, Outgoing TTL 255
Local host: 192.168.1.1, Local port: 646
Foreign host: 192.168.1.2, Foreign port: 41632

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x30844B0):
Timer Starts Wakeups Next
Retrans 1004 19 0x0
TimeWait 0 0 0x0
AckHold 980 926 0x0
SendWnd 0 0 0x0
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 0 0 0x0
DeadWait 0 0 0x0

iss: 3866498670 snduna: 3866524706 sndnxt: 3866524706 sndwnd: 4038
irs: 2412687425 rcvnxt: 2412712985 rcvwnd: 4092 delrcvwnd: 36

SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 364 ms, ACK hold: 200 ms
Flags: passive open, retransmission timeout, alias, gen tcbs
non-blocking reads, non-blocking writes

Datagrams (max data segment is 536 bytes):
Rcvd: 1972 (out of order: 0), with data: 983, total data bytes: 25559
Sent: 1978 (retransmit: 19), with data: 1001, total data bytes: 26035

sh ip traffic | in echo
111154 echo, 1 echo reply, 0 mask requests, 0 mask replies, 0 quench
Sent: 24728 redirects, 90828 unreachable, 0 echo, 111109 echo reply

show ip bgp ipv4 unicast neighbors 192.168.1.2 | in TTL - Nothing. No result.

show ip bgp ipv4 unicast neighbors 192.168.1.2 | sec Last reset

Last reset 11:52:26, due to BGP Notification received, no supported AFI/SAFI - This is strange as the configs have not changed at all yet suddenly stopped working. Could this be the reason?

[MHM Cisco World]

show ip bgp ipv4 unicast neighbors 192.168.1.2 | sec Last reset

Last reset 11:52:26, due to BGP Notification received, no supported AFI/SAFI - This is strange as the configs have not changed at all yet suddenly stopped working. Could this be the reason?

This can causes of flapping issue

@Harold Ritter please can make look I remember that there bug with sane conditions, can you confirm?

Thanks 

[CliveG]

Hi MHM

Unless this is a new bug I am a little confused. If we have not updated the IOS (same IOS as when working) and we have not changed any configuration why a bug would suddenly appear?

I can say that two weeks ago, due to no fault of ours, our P2P upstream supplier accidentally cut off both our upstream links from the Office DC to the 2 x Telehouse DCs. Given that we noticed this approximately 8 days ago, this is the only change that has occured. But surely just re-enabling a circuit will not have causaed this issue? (Noting that the issue appears to be beyond that disconnection point as well as including that disconnection point).

What this actually leaves us with, I think, is 3 things:

1: The IOS has somehow got corrupted - Therefore a hard reset may resolve this (and this will be the first troubleshooting step when on site, before swapping switches).

2: There is a hardware fault and it is not sfp related.

3: There is a bug (new).

As an add on, we have an upstream connection from the faulty unit to LoNAP and the 4 x peers there are also flapping.

[Harold Ritter]

Hi @MHM Cisco World ,

The issue seems to be related to BGP thinking it can send 7936 bytes as the TCP payload, but a ping between the two ibgp peers with a size of 7976 bytes (7936 + 20 for TCP header + 20 for IP header) is not allow. Since hello messages are embedded in update messages and that these large packets don't make it through, the session times out after 180 seconds.

It looks like something has changed recently in the underlay. 

It also looks like path mtg discovery is not working as it should, since it should be receiving ICMP packet too big messages and change the path mtu automatically.

The next step is probably to find out what is the maximum packet size supported between the two ibgp peers and to set the path mtu manually or to troubleshoot why the path mtu discovery does not work as it should.

Regards,

[MHM Cisco World]

Yes I read your post and Mr @CliveG  answer 
I solve one year ago same issue with NSK SW and the solution was 
1- disable MTU discovery
2- manually setting the MTU in both sides 
the MTU can find with extended ping and see until where the ping is stop.