7204VXR NPE-G1 Consistent Packet-Loss

[gordonmarquis]

Hello All...

We have an issue with two 7204VXRs with the same revision C NPE-G1 supervisor cards, which connect to one another over a circuit provided by our provider.

In pinging the other 7206VXR on the other end of the circuit we lose about 1 ping in every 700 or so, whether tagged as EF voice or natively.

However, if we source from loopback0 then repeat this test there is no packet loss!

Trouble-shooting steps completed:

1) Service provider has done two full circuit tests, which has shown no loss on the circuit

2) We have removed all QoS and user traffic from the circuit, packet-loss still same

3) Have worked with TAC in adding a shaper to our QoS policy, but this shaper gives inconsistent statistical output...TAC are at a loss, and as a last

    resort we will change the supervisor in each router soon. Seems to be a software bug/problem, not hardware in my opinion.

4) Our service provider has shown us graphs of our bandwidth usage, only generally 8to10% of the 100Mb circuit speed

5) gigabit controllers show no problem with ring size or output queue software drops.

6) There are "Total Output Drops" on the interface, but the increasing number does not match the level of loss, although it is regular too:

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2477

We had worked with TAC in changing the input and output queue hold sizes - no help. As mentioned, this circuit is not congested it would seem from both ourselves the the service provider's point of view.

7) Show controller gigabitX/X shows no drops or loss, etc

8) We have upgraded IOS, from latest T1 to T4: Both sides now on C7200-SPSERVICESK9-M), Version 12.4(24)T4

9) Router CPU usage around 10% or so

Coule we be bursting and this is not registering?

Ping loss is not huge:

Success rate is 99 percent (93230/93333), round-trip min/avg/max = 20/23/96 ms

But we are attempting to run other services, such as Cisco's circuit emulation product, over this circuit, and such regular drops cause major problems with the A and B end.

Short of testing this core circuit with other routers, has anybody heard of any bug or packet-loss related issue that could be causing what we see? I wonder why packet-loss is eliminated if we source from our loopback address when pinging - in any traffic class - yet pinging natively causes the packet-loss. The loss is real, and is noticable by other services running through; so it's not just the router dropping pings.

I do not trust the stats output on the 72XX platform, due to the way the platform is designed.

Thanks in advance for any input...

[Joseph W. Doherty]

"Realistically, how much can flow control delay packets?"

Honestly, I don't know. But I also don't really know what's its specific triggers (and releases) are or its pause intervals are either. I don't even know if egress congestion on the 7200 could trigger it on the ingress port.

That said, if you find it works for you, I see no reason not to use it, with the understanding if you don't know exactly how it's driven, a change in traffic behavior can come back to haunt you as, again, it's an all-or-nothing solution.

Also honestly, it's not something I've considered to address this situation. Usually, on a 7200, I've found "tuning" allows the 7200 to handle microbursts. If the traffic is more substained, I'll look to "shape" its upstream (which you also suggested - although I usually also take advantage of the upstream's QoS to manage "shaped" congestion). Using the pause feature (if supported), though, is very strait forward, and again, I would agree it could work in many situations just fine.

PS:

Oh, forgot go to add, yes indeed, please post how your increased ingress SPD impacts, if any, the drops.  (I've retired my last 7200 last month.)