1Gbps link, sweep ping droping packets

[Marcin Brzeski]

Hi, I hope someone can help.

We are testing a new 1Gbps WAN circuit between 2 sites. We have cisco 3750 and 4507 on each end.

Every time we run extended ping sweep ranging from 36 to 18024 bytes the packets are being dropped randomly once the size goes above 1500 bytes.

Our ISP claims Demark to Demark test are clean and they don't want to acknowledge the problem, they blame our switches. To prove the problem is not on our end we've put different switches at each end, still facing the same issue. Ping success rate is around 98 to 99 percent. Has anyone experienced such a problem? Thanks in advance!

[Federico Ziliotto]

Hi Marcin,

Maybe to help prooving to your ISP that the issue is in the MTU path between the two sites, you could run the ping tests while setting the DF bit to '1' and collecting a trace with Wireshark.

If the DF bit is set and the packet is dropped because it exceeds the MTU size, you should get back an ICMP destination unreachable  message with the "fragmentation needed and DF set" code.

Hope this helps,

Fede

--
If  this helps you and/or answers your question please mark the question as  "answered" and/or rate it, so other users can easily find it.

[cadet alain]

Hi,

the packets are being dropped randomly once the size goes above 1500 bytes. 

if this was a MTU issue wouldn't all packets be dropped once they're larger?

Regards.

Alain.

[shillings]

Yes, set the DF bit to 'no fragment' and run various Ping tests to work out your max unfragmented MTU - i.e.  ping x.x.x.x. size 1508 df-bit repeat 1000. If that works then increase the MTU. If not, drop it down to 1504 and so on.

Maybe it's not consistant due to different routing paths, but you need more information.

I presume there are no tunnels as that can impact Ping results when setting the df-bit?

Is the ISP P2P link over MPLS VPN or just off-net fibre between the two premesis?

What MTU does your ISP offer in their SLA? Maybe they only support 1500 Byte MTU anyway.

[Marcin Brzeski]

Thank you both for your replays,

Each side interface is at their default MTU 1500.

I have monitored the traffic with wireshark and the packets are being fragmented properly. It seems as after fragmentation the smaller packets are being dropped. Like I've said this is random but always above 1500 bytes.

We also have another 1 Gbps circuit on the same switch (currently in production) from Verizon and we have no problem doing the same sweep.

Thank you,

[cadet alain]

Like I've said this is random but always above 1500 bytes.

if indeed it is a problem with fragmented packets then it's normal it happens only above MTU.

It would be interesting to see if randomness is really randomness and where in the path are tey being dropped.

Regards.

Alain.

[shillings]

And your not tunneling? Any firewalls?

Just to confirm,

1500 bytes unfragmented is fine?

1501 unfragmented is always dropped?

1501 fragmented may drop the 2nd smaller packet?

[Marcin Brzeski]

Hi,

Yes, no tunnels, no firewalls.

Up to and 1500 bytes unfragmented is fine? - no problem

1501 unfragmented is always dropped? - always dropped, ISP does not allow anything biger than 1500

1501 fragmented may drop the 2nd smaller packet? - true

Thank you,

[shillings]

I suspect like Federico says, you'll have to prove to the ISP that the packet is leaving your router and not being received the other end. Can't think what could cause this other than a firewall getting the two fragments out of sequence.

[shillings]

Are you running NAT?

http://www.cisco.com/en/US/tech/tk648/tk361/technologies_tech_note09186a0080093f96.shtml

[Marcin Brzeski]

Hi shillings,

Interesting article but, no we do not run NAT it is a straigh P2P connection.

Thank you,

[Giuseppe Larosa]

Hello Marcin,

you say you would like to test the 1Gbps WAN link then you use sweep option over an extended size range.

Your test is also stressing the system buffers of the involved devices as the fragments have to be helded before re-assembly.

To see if this is the cause of this random losses I would suggest you, if you haven't already done, to increase the number of vty and to run the same test in two different conditions:

a single ping sweep session in a single VTY

multiple ping sweep session in multiple VTYs

if you are already using multiple sweep sessions, it is likely a buffer probem on the used devices.

the fact that with packets of size 1500 byte you have zero loss demonstrates that the link is operational over time.

What is more difficult to demonstrate is to check the real link bandwidth: it is really 1Gbps?

You should have two linux boxes  with 1GE NICs and to run an UDP performance test to say something but this is not always possible..

Hope to help

Giuseppe