3945 packet loss when passing traffic

[russell.sage]

I have a 3945 acting as an MPLS CE router. G0/0 is WAN port. G0/1 and G0/2 are connected to separate 3850 switches.

Identified packet loss on the WAN link. Investigation showed that if we closed the two LAN ports the packet loss stopped. IE a ping from the CE to PE router with LAN ports open showed packet loss. Close the LAN ports ping from CE to PE is error free. CE router was rebooted as where the two 3850 switches. No change.

Basic interface checks reveal nothing. CPU and memory all fine. Traffic levels are low <50Mbs. BGP and EIGRP are stable

Short of replacing of the box I am at a lost. Replacing the router is a logistical nightmare.

Any ideas welcome.

[Eric R.]

Hey Russell,

Do you mind elaborating on this a bit? Maybe I can help you get this sorted.

1. What version of IOS are you running?
2. Are you experiencing 100% traffic loss, or is some making it through?
   • If so, what is able to get through when you have your LAN ports down?
   • Is traffic dropping on the ingress or egress interface?

Depending on the version of code you're running, it's possible you could be hitting the following vulnerability ( cisco-sa-20160804-wedge ). Your issue sounds identical.

[russell.sage]

Hi

we are currently running 15.6(3)M6a

We are experiencing anywhere from 4-40% packet loss when the LAN ports G0/1 and/or G0/2 are open. close the ports and ping from the router to is BGP neigh is 100% error free.

Don't see any sign of packet drops on the routers interfaces.

As to the bug I don't think it affects us as we have ACL and CoPP in place. No changes have been made to the router config. There was a power is at the site about a week ago but this issue started over the weekend.

I can't see any indication of a hardware issue on the device. We have rebooted the router and the connected switches which has made no difference.

[Eric R.]

Well, this should be interesting. Let's see if we can try and figure out what it could be. I will have to do some digging, but I have folks I can reach out to if need be.

Would you be able to provide (and sanitize if need be) the following outputs?

• sh interface <int> (of the WAN interface and the two LAN interfaces)
• sh platform software drop-port
• sh processes cpu
• sh processes cpu history

Let's start by looking at these configs to see if anything is out of the ordinary. Just a couple of questions.

1. Are you running IPSec?
   1. If so, would be good for you to review the log ( sh crypto ipsec sa d | i internal ). Specifically, look at the internal error (received) packet count.
2. Were any devices on that network recently part of any sort of maintenance window (software upgrade/refresh/config change)?
3. Can you reproduce the issue on the spot? Or does it randomly resurface when you bring the LAN interfaces back up?

I am doing some research on my end, but I am down to a couple of possible causes. I just need to get a better understanding of the design. The fact that you're not seeing anything interrupting the routes makes me believe that one of my theories is correct. Right now, I am not sure if it is a bug or not as nothing is obvious, but hopefully we can see something in the configs.

[russell.sage]

hi

The LAN ports are currently shut as opening impacts the users. We have extensive SNMP based monitoring on the device. CPU and memory are good. I will look to get the outputs

1. Are you running IPSec?
   1. If so, would be good for you to review the log ( sh crypto ipsec sa d | i internal ). Specifically, look at the internal error (received) packet count. - no we are not running IPSEC
2. Were any devices on that network recently part of any sort of maintenance window (software upgrade/refresh/config change)?No
3. Can you reproduce the issue on the spot? Or does it randomly resurface when you bring the LAN interfaces back up? Yes as soon as we open the LAN ports we can reproduce the errors. 

[Joseph W. Doherty]

What's the physical port speed on G0/0 and does your contract, with your WAN vendor, provide physical port speed?

If you're contracted for a WAN bandwidth less than physical port speed, possibly WAN vendor is enforcing it.

If you do have full port speed, possible issue with WAN vendor.  If so, often you need to "prove" there's a problem on their side

[russell.sage]

Hi

G0/0 connects to a 100M LoS radio link back to the BGP neighbor router. I am the WAN vendor and the network management supplier. Solution has been in place the past 5years no significant changes. We had a significant rain storm prior to this issue starting and the radio alarmed during the storm but stopped once the storm stopped. My radio colleagues have checked and are happy with the radio. With ports 0/1 and 0/2 shut 1000 pings of 1400bytes runs to the BGP neighbor error free. Same test with the G0/1 and/or G0/2 ports open and we get packet loss anywhere from 4 to 40%. Can't see anything wrong with the router. CPU, memory, buffers temp all look good.

[Joseph W. Doherty]

Ok, radio link, upstream of 3945, 100 Mbps.  So g0/0 is physically running at 100 Mbps bandwidth?