Re: Issues Pinging Between Sites with Larger than Default Byte Size

Matthew Martin · ‎04-24-2019

Hello All,

There seems to be an issue when pinging between our HQ (*Site A - Firepower VM) and remote site (*Site B - SFR Module on ASA). Pinging is fine when using the default byte size. But, when pinging with, for example 500 bytes or larger, there's packets loss. And it seems fairly inconsistent going from Site A to B or reverse, I'll explain this below. It also seems ok, no matter the byte size, when pinging to and from each device's gateway.

For example, if I ping from A --to--> B with 500 bytes. Tcpdump on device B shows that maybe 60% of the ICMP Requests are even making it to B.

Then, if I ping from B --to--> A with 500 bytes, tcpdump on device A shows that ALL of the Requests are received by the site A device, but maybe only 60-70% of the Replies are making it back to device in site B.

Packet Flow from A to B:
Firepower VM > 4510 (*Edge Router) > MPLS > 2911 (*Edge Router) > 4507 > ASA/SFR Module

Any thoughts or suggestions would be greatly appreciated!

Thanks in Advance,
Matt

mlund · ‎04-25-2019

Hi

Can You try with some other test, that is tcp or udp based. It can be some restriction in the path for icmp packets.

/Mikael

Matthew Martin · ‎04-26-2019

Hi Mikael, thanks for the reply.

Do you have any suggestion on what kind of test I can do?

Here's an example pinging the opposite device's gateway:

Ping from Device A to device B's gateway:

admin@firepower:~$ sudo ping -c 20 -s 500 10.50.123.1
Last login: Fri Apr 26 16:58:12 UTC 2019 on pts/1
PING 10.50.123.1 (10.50.123.1) 500(528) bytes of data.
508 bytes from 10.50.123.1: icmp_req=1 ttl=251 time=40.4 ms
508 bytes from 10.50.123.1: icmp_req=2 ttl=251 time=41.7 ms
508 bytes from 10.50.123.1: icmp_req=3 ttl=251 time=1277 ms
508 bytes from 10.50.123.1: icmp_req=5 ttl=251 time=3645 ms
508 bytes from 10.50.123.1: icmp_req=10 ttl=251 time=2987 ms
508 bytes from 10.50.123.1: icmp_req=13 ttl=251 time=4337 ms
508 bytes from 10.50.123.1: icmp_req=18 ttl=251 time=3709 ms

--- 10.50.123.1 ping statistics ---
20 packets transmitted, 7 received, 65% packet loss, time 19108ms
rtt min/avg/max/mdev = 40.421/2291.419/4337.767/1676.405 ms, pipe 5

Ping from Device B to device A's Gateway:

admin@SFR3:~$ sudo ping -c 20 -s 500 192.168.2.1 
Last login: Fri Apr 26 16:58:40 UTC 2019 on pts/1
PING 192.168.2.1 (192.168.2.1) 500(528) bytes of data.
508 bytes from 192.168.2.1: icmp_req=1 ttl=249 time=32.1 ms
508 bytes from 192.168.2.1: icmp_req=2 ttl=249 time=32.0 ms
508 bytes from 192.168.2.1: icmp_req=3 ttl=249 time=30.8 ms
508 bytes from 192.168.2.1: icmp_req=4 ttl=249 time=29.5 ms
508 bytes from 192.168.2.1: icmp_req=5 ttl=249 time=32.1 ms
508 bytes from 192.168.2.1: icmp_req=6 ttl=249 time=30.9 ms
508 bytes from 192.168.2.1: icmp_req=7 ttl=249 time=65.9 ms
508 bytes from 192.168.2.1: icmp_req=8 ttl=249 time=29.9 ms
508 bytes from 192.168.2.1: icmp_req=9 ttl=249 time=34.4 ms
508 bytes from 192.168.2.1: icmp_req=10 ttl=249 time=45.8 ms
508 bytes from 192.168.2.1: icmp_req=11 ttl=249 time=36.9 ms
508 bytes from 192.168.2.1: icmp_req=12 ttl=249 time=35.3 ms
508 bytes from 192.168.2.1: icmp_req=13 ttl=249 time=33.0 ms
508 bytes from 192.168.2.1: icmp_req=14 ttl=249 time=44.6 ms
508 bytes from 192.168.2.1: icmp_req=15 ttl=249 time=47.9 ms
508 bytes from 192.168.2.1: icmp_req=16 ttl=249 time=38.4 ms
508 bytes from 192.168.2.1: icmp_req=17 ttl=249 time=41.9 ms
508 bytes from 192.168.2.1: icmp_req=18 ttl=249 time=44.4 ms
508 bytes from 192.168.2.1: icmp_req=19 ttl=249 time=39.1 ms
508 bytes from 192.168.2.1: icmp_req=20 ttl=249 time=42.9 ms

--- 192.168.2.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19019ms
rtt min/avg/max/mdev = 29.580/38.447/65.910/8.508 ms

-Matt

Matthew Martin · ‎04-26-2019

Also, another note...

If I increase the interval between each ping packet that's sent, the packet loss is reduced. For example, from 65% loss in my previous comment using the default 1 second interval, down to 15% using 5 second interval...

Not sure if that helps, but thought I would mention it.

-Matt

mlund · ‎04-29-2019

Hi

when pinging from B to A:s gw you actually get the full path, but not the physical interface where A is connected, and as this is going fine, I would suggest to look closer on the path from A (including A) through switches (if thera are any) and the physical interface of A gateway.
Is there any errors or are there any configuration that maybe is restriction traffic. Some mls ratelimiting or a policy-map maybe.
For other tests you can use nping for example, it can sen traffic with udp or tcp, download it from nmap.

/Mikael

Matthew Martin · ‎04-30-2019

Hey Mikael, thanks for the reply.

So Device A is actually a Cisco Firepower virtual machine. So its housed in UCS. The interface where that is connected is TenGigabitEthernet5/1, which is connected via fiber.

Interface on 4510R:

interface TenGigabitEthernet5/1
 description UCS 6248A
 switchport mode trunk
 switchport nonegotiate
 channel-group 18 mode active
end
!
!
interface Port-channel18
description UCS-6248XP A
switchport
switchport mode trunk
switchport nonegotiate
flowcontrol receive on
end

Below is the interface that connects Site A to our MPLS (*between site A and B).

interface GigabitEthernet1/1
 description MPLS
 no switchport
 bandwidth 102400
 ip address ********** 255.255.255.252
 load-interval 30
 speed 100
 duplex full
 auto qos trust
 service-policy input AutoQos-4.0-Input-Policy
 service-policy output AutoQos-4.0-Output-Policy
end

I will try and do some tests with nping and report back what I find.

Thanks Again,

Matt

Matthew Martin · ‎04-30-2019

Another Note... This below leads me to believe the issue is with the Firepower VM itself.

I gathered up all the host IP Addresses for all VMs located on Vlan2 (*192.168.2.x), and then I did test pings to all of these devices, from location B's gateway. About 15 total VM hosts on that Vlan. All of these pings go over the EXACT same path/wire to Te5/1 where UCS is located. And the ONLY VM to experience any packet loss when using a larger then default byte size, was the Firepower VM.

Would you say this is pretty conclusive that the issue is with the VM itself?

Thanks Again,
Matt

mlund · ‎05-01-2019

Yes, It seems pretty shure that it is the actual Vm that is facing problem.

But, I saw You have a service-policy input and output on the interface, to be sure, can You examine what that service-policy is actually doing, there might be a small chans that it is this policy that is restricting traffic.

/Mikael

Matthew Martin · ‎05-02-2019

What commands should I run to check out this Policy?

-Matt