ME3600 packet loss problem - reboot to fix.

[paolo_carbone]

One of our ME3600 units has developed a problem where we would lose IPv6 Neighbours and OSPFv3 neighbours, followed by the device not passing MPLS traffic. IPv4 traffic suffering bad packet loss.

I discovered that devices attached to the Gigabit ports were receiving errored frames so the ME3600 seemed to be sending out bad frames.

rebooting the device would fix it for a while but it kept happening at regular intervals, usually when the box was carrying more ipv6 traffic.

We have upgraded to the latest firmware which did not help, and performed a full hardware replacement from Cisco only to find we are still seeing the same issue!

there was no configuration change that I could point as being a catalyst for the problem, the most recent thing was enabling ipv6 VRFs, but IPv6 is also carried in the global table while we move some services around.po

Has anyone else experienced similar problems with the ME3600?

[brwbailey · Accepted solution]

This sounds very similar to a problem we have seen across multiple boxes.

I've not noticed whether OSPFv3 neighcors drop, however have seen MPLS stop forwarding and heavy v4 loss.  Control plane for v4 seems to stay up (ie v4 OSPF neighbors keep their adjacany) so routes still advertised etc.  Performing a shut / no shut of one of the core interfaces will generally result in a complete failure of the interfaces on that ASIC (the two 10G ports in our case) - can't ping adjacent devices after the no shut, but the GE ports still work.

10G ports are core facing on our boxes, and adjacent neighbors see increasing input errors when the issue occurs.

Issue has only been seen during peak traffic times.  Reboot fixes the problem.

Did you find any resolution?  We've been pointed at CSCui23725, and advised to use the command "platform acl egress-disable".  This appears to be working so far.

[paolo_carbone]

Hi

we have lowered the amount of traffic traversing the unit and it has stopped breaking so often, but we still do not have a resolution.

we have been sending all manner of logs and information into TAC but they are still working on it.

thanks for your comments, I will ask about "platform acl egress-disable" and see what they say.

[paolo_carbone]

Hi,

We implemented "platform acl egress-disable" and have not had any further problems with the device, so it looks like this may well have been the workaround for the bug.

Thanks for your help, our Cisco TAC contacts didn't offer this as a solution and were wanting us to continue capturing information each time it broke so they could resolve the problem, but with live services we had to perform the work around.

For other people's reference we started to have problems after we activated IPv6 in VRF.

[brwbailey]

This sounds very similar to a problem we have seen across multiple boxes.

I've not noticed whether OSPFv3 neighcors drop, however have seen MPLS stop forwarding and heavy v4 loss.  Control plane for v4 seems to stay up (ie v4 OSPF neighbors keep their adjacany) so routes still advertised etc.  Performing a shut / no shut of one of the core interfaces will generally result in a complete failure of the interfaces on that ASIC (the two 10G ports in our case) - can't ping adjacent devices after the no shut, but the GE ports still work.

10G ports are core facing on our boxes, and adjacent neighbors see increasing input errors when the issue occurs.

Issue has only been seen during peak traffic times.  Reboot fixes the problem.

Did you find any resolution?  We've been pointed at CSCui23725, and advised to use the command "platform acl egress-disable".  This appears to be working so far.