cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
752
Views
3
Helpful
7
Replies

why 10Gbps circuit every 12 packets drop 1packet

Herman2018
Level 3
Level 3

Hi, we have nexus in one side and C at another side,, and now try to provision one 10gbps point-to-point circuit, now the link is up, but when do ping test, found every 12 packets drop 1 packet.  mtu is set to 15xx at both ends. can anyone help advise? if use small packet to ping (56byte), then about 0.4% packet loss, pls advise, thanks in advance.

NxK#ping 1.1.1.1 count 1000 packet-size 15xx df-bit

...

Request 951 timed out
1508 bytes from 1.1.1.1: icmp_seq=952 ttl=254 time=1.035 ms
1508 bytes from 1.1.1.1: icmp_seq=953 ttl=254 time=0.867 ms
1508 bytes from 1.1.1.1: icmp_seq=954 ttl=254 time=0.86 ms
1508 bytes from 1.1.1.1: icmp_seq=955 ttl=254 time=0.864 ms
1508 bytes from 1.1.1.1: icmp_seq=956 ttl=254 time=0.979 ms
1508 bytes from 1.1.1.1: icmp_seq=957 ttl=254 time=0.866 ms
1508 bytes from 1.1.1.1: icmp_seq=958 ttl=254 time=0.851 ms
1508 bytes from 1.1.1.1: icmp_seq=959 ttl=254 time=0.843 ms
1508 bytes from 1.1.1.1: icmp_seq=960 ttl=254 time=0.926 ms
1508 bytes from 1.1.1.1: icmp_seq=961 ttl=254 time=0.846 ms
1508 bytes from 1.1.1.1: icmp_seq=962 ttl=254 time=0.921 ms
Request 963 timed out
1508 bytes from 1.1.1.1: icmp_seq=964 ttl=254 time=1.044 ms
1508 bytes from 1.1.1.1: icmp_seq=965 ttl=254 time=0.871 ms
1508 bytes from 1.1.1.1: icmp_seq=966 ttl=254 time=0.864 ms
1508 bytes from 1.1.1.1: icmp_seq=967 ttl=254 time=0.838 ms
1508 bytes from 1.1.1.1: icmp_seq=968 ttl=254 time=0.902 ms
1508 bytes from 1.1.1.1: icmp_seq=969 ttl=254 time=0.849 ms
1508 bytes from 1.1.1.1: icmp_seq=970 ttl=254 time=0.854 ms
1508 bytes from 1.1.1.1: icmp_seq=971 ttl=254 time=0.897 ms
1508 bytes from 1.1.1.1: icmp_seq=972 ttl=254 time=0.897 ms
1508 bytes from 1.1.1.1: icmp_seq=973 ttl=254 time=0.91 ms
1508 bytes from 1.1.1.1: icmp_seq=974 ttl=254 time=0.95 ms
Request 975 timed out
1508 bytes from 1.1.1.1: icmp_seq=976 ttl=254 time=0.977 ms
1508 bytes from 1.1.1.1: icmp_seq=977 ttl=254 time=0.871 ms
1508 bytes from 1.1.1.1: icmp_seq=978 ttl=254 time=0.874 ms
1508 bytes from 1.1.1.1: icmp_seq=979 ttl=254 time=0.874 ms
1508 bytes from 1.1.1.1: icmp_seq=980 ttl=254 time=0.947 ms
1508 bytes from 1.1.1.1: icmp_seq=981 ttl=254 time=0.893 ms
1508 bytes from 1.1.1.1: icmp_seq=982 ttl=254 time=0.872 ms
1508 bytes from 1.1.1.1: icmp_seq=983 ttl=254 time=0.917 ms
1508 bytes from 1.1.1.1: icmp_seq=984 ttl=254 time=0.901 ms
1508 bytes from 1.1.1.1: icmp_seq=985 ttl=254 time=0.871 ms
1508 bytes from 1.1.1.1: icmp_seq=986 ttl=254 time=0.899 ms
1508 bytes from 1.1.1.1: icmp_seq=987 ttl=254 time=0.914 ms
Request 988 timed out
1508 bytes from 1.1.1.1: icmp_seq=989 ttl=254 time=1.404 ms
1508 bytes from 1.1.1.1: icmp_seq=990 ttl=254 time=0.879 ms
1508 bytes from 1.1.1.1: icmp_seq=991 ttl=254 time=1.023 ms
1508 bytes from 1.1.1.1: icmp_seq=992 ttl=254 time=0.902 ms
1508 bytes from 1.1.1.1: icmp_seq=993 ttl=254 time=0.87 ms
1508 bytes from 1.1.1.1: icmp_seq=994 ttl=254 time=0.866 ms
1508 bytes from 1.1.1.1: icmp_seq=995 ttl=254 time=0.968 ms
1508 bytes from 1.1.1.1: icmp_seq=996 ttl=254 time=0.908 ms
1508 bytes from 1.1.1.1: icmp_seq=997 ttl=254 time=0.896 ms
1508 bytes from 1.1.1.1: icmp_seq=998 ttl=254 time=0.9 ms
1508 bytes from 1.1.1.1: icmp_seq=999 ttl=254 time=0.91 ms

--- 1.1.1.1 ping statistics ---
1000 packets transmitted, 921 packets received, 7.90% packet loss
round-trip min/avg/max = 0.803/4.963/87.305 ms

1 Accepted Solution

Accepted Solutions

balaji.bandi
Hall of Fame
Hall of Fame

check on the nexus any Control plane polices applied ?

what is the nexus side IP address ?

instead of using MTU can you just ping using source interface from nexus with count and what is the outcome ?

 

Note : 1.1.1.1 is public routable IP.

 

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

View solution in original post

7 Replies 7

balaji.bandi
Hall of Fame
Hall of Fame

check on the nexus any Control plane polices applied ?

what is the nexus side IP address ?

instead of using MTU can you just ping using source interface from nexus with count and what is the outcome ?

 

Note : 1.1.1.1 is public routable IP.

 

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Thanks @balaji.bandi  for your kind advice. This is control policy . 

Not surprising a control plane policy policing pings.

What is surprising (to me, anyway), it is being so regular, i.e. every 12th packet.

@Joseph W. Doherty Hey, Joe. Control plane policers are typically implemented using token-bucket policers in the NPU (police the punted packets in h/w before they hit s/w in the CPU). The regularity of the dropped packet by the policer indicates it takes 11 packets to drain the tokens in the bucket below the ping packet size, as the bucket is being refilled with tokens at a slower rate than they are being consumed by the ping packets.

When the 12th packet arrives, there are not enough tokens for the policer to pass it to the CPU, so it is discarded. The discard causes a timeout in the ping sender, which provides enough time for the token bucket to be completely refilled, resulting in the cycle starting over again with the 13th ping packet. 

I used to see similarly predictable behavior in working with a mobility operator in tests of their configurations for control plane policers. 

Disclaimer: I am long in CSCO

(@Ramblin Tech) Jim, thank you for describing control plane policing, I'm "somewhat" conversant on that, but it's likely to be beneficial to other readers.

FYI, why I was surprised this being so regular, i.e. every 12th packet (which might not be totally true), because there are many possible variables that could impact this in a production environment.

For starters, look at the OP ping stats times, i.e.:

round-trip min/avg/max = 0.803/4.963/87.305 ms

Better than a 100x spread between min and max.

Whenever Cisco devices are processing ping requests, it seems to be processed in software (i.e. control plane) as a low priority task.  So much so, Cisco provided RTR (now part of IP SLA), to timestamp arrival of a ping request, so that delay time, sitting on the responding Cisco device, can be mitigated to allow a much more accurate network RTT.

There might also be possible timing issues of how quickly a host transmits the ping requests.  (I.e. does pinging program/process send each request back-to-back, or provide some inter ping request pause?  [NB: I've noted, when using a long series of ping requests, utilization stats appears higher as you increase ping packet's size, so either there's some pause between ping requests or lower layer overhead isn't being counted.])   Sending host might be interleaving other network traffic for transmission, along with, along the path, the ping requests might be interleaved with other traffic from other hosts.

I.e., because of the above, it's a bit uncertain exactly when each ping request arrives to the host, which could, impact the policer's Tc usage computations.

Possibly, there are relevant considerations about exactly what the CoPP is doing (also possibly, platform specific).  For example, is CoPP running an on-going policer timer or does it only start one with the first packet that triggers policing?  This can impact how the actual traffic aligns with the policer's Tc boundaries.  (NB: should be less of a factor the longer the series of packets is.)

(As example of the above, assume you send a train of packets, That consume just under 100% of two Tcs.  If the Tc accounting starts with the first packets, the first Tc will be "over rate" and packets will be dropped.  But if first packets arrives just a tad in from the beginning of the first Tc, and the last packet doesn't "overflow" the second Tc, then that packet train will not be impacted.  [BTW, this is often why "default" traffic policers tend to provide an actual transmission rate than configured for.])

Also how does the policer account for packets that span a Tc boundary?

What exactly is the policer counting for Tc usage?  I.e. packet's bits, frame's bits or actual wire consumption (the latter accounting for Ethernet preamble and SFD).  (Oh, also take note how OP reports a 0.4% lost for 56 byte packets, but stats for 1560 byte packets show a 7.9% drop rate.  Why the greater than 10x loss rate disparity?  So, possibly, some of the possible variables I mention, do make a difference.)

The above issues, I suspect, might cause some "drift" in whether it's always the 12th packet that gets dropped.  (Oh, and as to the importance of some of these considerations, consider why some Cisco platforms provide more "advanced" bandwidth consumption algorithms, see, for example, Cisco's deficient round robin queuing variants vs. variants that don't do similar.)

So, in summary, except perhaps in a lab, I would have expected some variance in every X numbered packet being dropped, but it's possible the issues, above, might also have less impact on the larger ping packets being used (another possible issue, are packets being fragmented sent as size 1560?  [as DF is set in OP, we might assume they are not, but as DF can be ignored, is it?]) or "every" isn't truly always every.

Regardless, to OP, @balaji.bandi identified the issue.  But, whenever there's "issues" with pinging, (especially) Cisco network devices arise, I often mention, it's my understanding, ping was intended mostly as a basic tool to determine if another host is "alive".  I.e. unexpected results might not represent any "true" network or host issue.

Joe, astute observations, as usual. I am surmising that every 12th packet being dropped is actually a nominal average, since the example above, on closer examination, actually shows the 13th packet being dropped in the case of #988! For all the reasons you mentioned, there will be some variability in each RTT leading to some variability in the number of tokens being available for consumption in the bucket (tokens being added at a constant rate, but being consumed at a slightly varying rate).

Disclaimer: I am long in CSCO


@Ramblin Tech wrote:

. . . the example above, on closer examination, actually shows the 13th packet being dropped in the case of #988!


Ah, I missed seeing that!

Yes, I agree with "I am surmising that every 12th packet being dropped is actually a nominal average . . .", which is different from EVERY 12th packet.

Review Cisco Networking for a $25 gift card