Re: counters on ISR. bgp tshooting

Amafsha1 · ‎04-01-2018

Hello, I have an ISR 4000 series router with image Cisco IOS XE Software, Version 16.03.06
Cisco IOS Software [Denali], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.6, RELEASE SOFTWARE (fc3).

When executing the command "show int counters" and "show int counter errors" I get the error message of "Counters information is not available for GigabitEthernet0/0/3". This is really frustrating as I need to see the counter errors from what's believed to be a bad issue with the sfp gbics and/or the fiber optic cabling. (this is a router that connects to the fiber handoff at the demarc) Not sure what to do here because the ISP is saying we have bad optics on our side because they tested their side. Does anyone have a recommendation of a command I can run to get information about physical problems? or what log I can set to report on that?

what makes this matter worse is our equipment never prints any logs of any sort of issue, and we randomly loose BGP neighborship with the ISP router, but the interface never goes down and doing a "clear ip bgp" doesen't do the trick, we have to shut/noshut the interface to get the neighborship to form again. The only 1 semi-usefull thing I found was everytime we lost BGP neighborship with ISP router, the "lost carrier" counter would increment by a couple.

cofee · ‎04-02-2018

Hi,

Not sure why it's not showing any output with those commands, but could be happening due to a bad interface. Have you tried replacing gbic or using another port on the router? I would assume if it's bad fiber optic then you should have seen runt/crc error counters going up on the Ethernet port.

We had experienced similar issue with our isp several months ago where bgp peering will be torn down randomly and when we raised this issue with the isp we were told they didn't find anything on their end. It wasn't big of a deal since we had a redundant connection and all traffic would fail over, but it ended up to be a faulty module on the isp side and they only discovered that when it crashed. That faulty module generated enough warning signals before it crashed but they failed to catch it in their troubleshooting.

While your resolve this issue I would suggest to use an eem script to shut/no shut the interface when it detects bgp peering is down, so you guys won't have to manually do it and outage will be minimal.

Amafsha1 · ‎04-03-2018

Thanks for writing back. Yeah, we're essentially playing the blame game with the ISP, we replaced sfp, swapped out port etc.. The only piece of information they can give us is "we have a LOS of 8 seconds from your end". I would think that if we experienced a true LOS on our side, our interface would shutdown and warn us...but not. The first log that appears for us every single time is:

%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired) 0 bytes

That's the best piece of information I have. All that's left for us to do is replace the router.

Do you think you can possible give me a summary of how your failover works? like very basic high level configs? we're trying to set something like that up.

cofee · ‎04-03-2018

Sure, it pretty straightforward. We have 2 edge routers and each is connected to separate WAN providers. Currently, we are not doing any load balancing between these 2 circuits and use them as active/standby. Both routers are connected to each other via an Ethernet port with hsrp configuration, hsrp is configured with tracking of WAN interface, if the WAN interface goes down then hsrp kicks in and reduces its priority on the port which is hsrp aware, so the standby router can take over.

The issue that you are currently experiencing was giving us trouble, because the serial interface would still be up but BGP peering will tear down and since the interface stayed up hsrp won't kick in. We had to write an eem script on the affected router to shut down the WAN interface when bgp peering is down so failover can take place automatically.

Amafsha1 · ‎04-03-2018

Great configuration. I have 2 questions:

1. Let's say the interface encounters slowness, do you guys just manually flip the circuit?

2. Why don't you loadbalance?

cofee · ‎04-04-2018

1. Let's say the interface encounters slowness, do you guys just manually flip the circuit?

We don't have such policy because it was never needed. But this sort of policy would probably need to monitor circuit utilization. We do have a policy where router tracks WAN circuit crc errors and if it hits a certain threshold, it fails over to back up link.

2. Why don't you loadbalance?

Customer doesn't want loadbalace.

Amafsha1 · ‎04-10-2018

So since you have hsrp as the failover mechanism for tracking your outside interface, don't you also need to run iBGP between your 2 edge routers? What would happen if that hsrp interface would go down?

I guess it wouldn't make sense to run iBGP because you're connecting to 2 different bgp.