Troubleshooting Packet Loss

[anthonypoon]

Hi experts,

We have difficulties to troubleshoot our intermittent call drop issue at one remote site (remote site 1). IPT was deployed to this site recently with IP Phone 7821 and C9300. The users said call drop happens when callling to specific office (remote site 2).  We found both peer are IP hone 7821. We have upgraded 7821 firmware, replaced network switch and SCCP IP phone but in vain. Our IPT vendor captured packet and observe there should be something wrong in host site 1 network (please refer to the attachment). 

Can you please advise how to easily troubleshooting and finding out solution for this case? We have some limitations on our infrastructure and business nature. 

Thanks.

[Leo Laohoo]

Post the complete output to the command "sh interface <WAN port>" of remote site 1.

[anthonypoon]

Hi,

I posted our 3 routers' interface information, fa0/0/0 is the WAN interface. Thx.

[Leo Laohoo]

I'm not opening any RTF in a zip file.
Post the RTF files here.

[anthonypoon]

Hi,

Here you are. Thx.

[Leo Laohoo]

What is RTR1? Look at Fa0/1. Look at the huge amount of "input errors". That's where the problem is. Look at the ratio between the input packets and the "input error". Now look at the "Full-Duplex" and "100Mb/s".
The cause is due to duplex mismatch error.
Look at the config for Fa 0/1 and see if duplex and speed are hard-coded. If it is, take them off.
NOTE:
I'm actually surprised that voice was their only complaint.  EVERYTHING would've been slow.  Everything.

[Deepak Kumar]

Hi,

RTR1, RTR2, and RTR3 are showing input or output drops as RTR2 and RTR3 are having gig port but it is working on FastEthernet. I am not sure that it is your next hope hardware limitation or manually it assigned to 100-Full Duplex. Is it speed and duplex mismatch? Have you applied any QoS?

Other drops are looking as "Unknown Protocol Drop" but currently, I am not considering as critical because this port may be connected to the router/switch and remote device will not understand many protocols as DTP, VTP, etc. 

[anthonypoon]

Hi,

Thanks for your hints. Our carrier has correct the interface speed & duplex mismatch issue and there is no more huge input error. However, our remote site 1 user are still complaining call drop.

I just did a deeper trace - I found packet loss even ping from Remote Site 1 switch to the IP phone x8081 but the interface gi1/0/12 cable test is normal. I think there is something wrong in Remote Site 1. What is your thought?

When pinging from Host Site 1 to Remote Site 2 phone x8084, there was no packet loss.

Protocol [ip]:

Target IP address: 10.104.116.106

Repeat count [5]: 5000

Success rate is 100 percent (5000/5000), round-trip min/avg/max = 1/2/12 ms

Tracing the route to 10.104.116.106

VRF info: (vrf in name/id, vrf out name/id)

  1 10.115.85.6 0 msec 0 msec 4 msec

  2 172.24.249.11 4 msec 0 msec 0 msec

  3 10.114.116.21 12 msec 0 msec 4 msec

  4 10.104.116.106 4 msec 4 msec 0 msec

When pinging from Host Site 1 to Remote Site 1 x8204, there was one packet loss.

Protocol [ip]:

Target IP address: 10.104.113.110

Repeat count [5]: 5000

Success rate is 99 percent (4999/5000), round-trip min/avg/max = 1/2/204 ms

When pinging from Host Site 1 to Remote Site 1 phone x8081, there was many packet loss.

Protocol [ip]:

Target IP address: 10.104.113.124

Repeat count [5]: 5000

!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.

!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!.

!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!

!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!

!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!

.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!

.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!!

!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!

!.!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!.

!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!

!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!

!!!.!!!!!!!!.!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!!!!!!.!!

!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.

!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!

!!.!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!

.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!

!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!

!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!

!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!

.!!!!!.!!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!

!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!

!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!!.!!!!!.!!!!!!!!

.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!

.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!

!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!

!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!

!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!

!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!

!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!

!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!

!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!.!

!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!.!!!!

!.!!!!!.!!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.

!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!

!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!

!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!

!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.

!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!

!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.

!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!

.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!

!.!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!

!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.

!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.

!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!

!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!

!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!!!!!!.

!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!

.!!!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!

.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!

.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.

!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!

!.!!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!

.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!

!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!!

!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!

!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.

!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.

!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!

!!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.

!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!

!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!

!!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!.!!!!!.!!!

!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!.!!!!!.!!!!!.!!!!!!!.!

!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!.

!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!

!!!!.!!!!!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!

!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.!

!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!

!.!!!!!!!.!!!!!.!!!!!.!!!!!.!!

Success rate is 86 percent (4305/5000), round-trip min/avg/max = 1/2/28 ms

When pinging from DC ROUTER to Remote Site 1 phone x8081, there was many packet loss.

Protocol [ip]:

Target IP address: 10.104.113.124

Repeat count [5]: 5000

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 5000, 100-byte ICMP Echos to 10.104.113.124, timeout is 2 seconds:

!!!!!.!!!!!.!!!!!.!!!!!!!.!!!!!.

Success rate is 84 percent (27/32), round-trip min/avg/max = 1/2/4 ms

When pinging from Remote Site 1R OUTER to Remote Site 1 phone x8081, there was many  packet loss.

Protocol [ip]:

Target IP address:

% Bad IP address

Remote Site 1-A-1811-R1#

Remote Site 1-A-1811-R1#PING

Protocol [ip]:

Target IP address: 10.104.113.124

Repeat count [5]: 5000

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 5000, 100-byte ICMP Echos to 10.104.113.124, timeout is 2 seconds:

!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!

!!!!.!!!!!.!!!!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.!!!!!.

Success rate is 84 percent (106/126), round-trip min/avg/max = 1/1/4 ms

It seems that there are packet loss from DC to Remote Site 1.

When pinging from Remote Site 1 SWITCH to Remote Site 1 phone x8081, there was many packet loss.

Remote Site 1-A-3560-1>en

Password:

Remote Site 1-A-3560-1#ping

Protocol [ip]:

Target IP address:

% Bad IP address

Remote Site 1-A-3560-1#ping

Protocol [ip]:

Target IP address: 10.104.113.124

Repeat count [5]: 5000

Datagram size [100]:

Timeout in seconds [2]:

Extended commands [n]:

Sweep range of sizes [n]:

Type escape sequence to abort.

Sending 5000, 100-byte ICMP Echos to 10.104.113.124, timeout is 2 seconds:

!!!!!!!!!!.!!!!!!!!!!.!!!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!!!!!!.!!!!!.!!!

!!!!.!!!!!.!!!!!!!!.!!!!!!!!!!.!!!!!!!!!!.!!!!!.!!!!!.!!!!!!!!.!!!!!!!

!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!!!!!!.!!!!!.!!!!!!!!!!.

Success rate is 88 percent (173/195), round-trip min/avg/max = 1/3/10 ms

Cabling test Normal to interface 1/0/12 (where Remote Site 1 phone X8081 connecting)

TDR test last run on: October 11 20:13:45

Interface Speed Local pair Pair length        Remote pair Pair status

--------- ----- ---------- ------------------ ----------- --------------------

Gi1/0/12  100M  Pair A     0    +/- 1  meters Pair B      Normal

                Pair B     0    +/- 1  meters Pair A      Normal

                Pair C     33   +/- 1  meters N/A         Short/Crosstalk

                Pair D     33   +/- 1  meters N/A         Short/Crosstalk

[Leo Laohoo]

Wait ... That doesn't make any sense.
The phone is directly connected to the switch. And if the switch pings the phone there are packet drops?
No way. That really doesn't make any sense.

[Deepak Kumar]

Hi,

As I am understanding that there is a drop on the local site network as Gateway to Phone. This means something is not good in the local LAN at the remote site. Is there any drop from the switch to Phone?

Can you confirm a few things:

1. Any CPU/memory high utilization of router/switch?
2. Any interface issue or cable issue? Can you replace cable from the switch to the phone or router to switch? Once, We noticed in my office that cable testing was showing no issue with the cable but flood testing was shown a huge drop in the singles due to one IO-BOX installation method. So, try to check with connect a phone directly to the switch.
3. Is your Phone 10/100 or 1000 Mbps? If your switch is  1Gbps and phone support to 100Mbps then check with speed and duplex issue?
4. Any POE issue?
5. Any routing or interface or Qos configuration issue for VOIP VLAN?