Packet loss in a Trunk

[naveen98]

Hi team,

We have an ongoing issue with our customer connected CCT bundle, and it is showing packet drops. We were able to isolate the issue to a single point and the issue is between Provider Edge switch to Cat 4500 cluster. We have connected a test router into the same CCT bundle by assigning an extra port (Gi 1/1/7). From that router to VLAN interface 825, we are observing packet drops. 

Troubleshooting Steps (Provider edge SW to CAT 4500)

1. First, we have changed the cables 2 times. Still, the packet drops can be monitored.
2. Then we have changed the port and SFP to Ethernet transceiver (This is a copper link connected to SFP port) yet no improvements. 
3.  We have changed the port and Ethernet transceiver on Provider edge SW but no luck.

Please find the trunk interface Te1/2/5 configurations

interface TenGigabitEthernet1/2/5
description *** Connected to XXXXXXX ***
switchport trunk allowed vlan 802-834,836-849
switchport mode trunk
mtu 9170
end

We have more than 20 customers connecting to our site via this CCT bundle and all of them facing this packet drop. All 20 plus customers have separate VLAN interfaces and all of them reside in a single VRF.

As the last troubleshooting step, we are to change the  medium from copper to fiber. Can I get Can I get any wisdom from the community on this issue? 

1. 

[balaji.bandi]

show us the interface output , what IOS code running on the switches.

you mentioned 1/2/5 and you posted the different interface config ?

does the MTU matches both the side ? (is your provide switch support that MTU ?)

if this VSS - which one is Active Switch ? - have you tried different cable to other switch (part of the VSS ?)

where is your spanning tree root for that VLAN ?

[naveen98]

Hi Balaji,

Switch Software : Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.11.05.E RELEASE SOFTWARE (fc3)

1/2/5 and 2/2/5 are the interfaces on 2 switches. Currently, traffic is going via 2/2/5 . I have mistakenly shared the config of the testing interface, which was same as 1/2/5. 

SW-MALI-MOB03#show interfaces Te2/2/5
TenGigabitEthernet2/2/5 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 5c83.8ff5.f874 (bia 5c83.8ff5.f874)
Description: *** Connected to SLT_Corporate backup ***
MTU 9170 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 18/255, rxload 11/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseT
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:13, output never, output hang never
Last clearing of "show interface" counters 00:03:42
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 46962000 bits/sec, 15662 packets/sec
5 minute output rate 71299000 bits/sec, 16974 packets/sec
3371915 packets input, 1161585669 bytes, 0 no buffer
Received 543 broadcasts (267 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
3626422 packets output, 1800277657 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Different cable was tested for the other switch, also we have tested a fiber cable as well. Still, the packet drop is visible. Cat switch is the root bridge for all the VLANS

[DanielP211]

Hello!

Packet loss can be due to multiple issues. Could you please first display the following commands (check for output/input erros, overruns, CRC):
sh int TenGigabitEthernet2/1/5

You can also enable history BPS output-drops and check the interface later on. If you have issues with packet queing else you can enable flowcontrol. If there are erros/CRC you have an issue with the cable/optical...

BR

[Joseph W. Doherty]

What kind of drops?  I.e. what do the interface stats show?

[naveen98]

When we do 1000 pings from test router to Cat 4500 SW L3 VLAN interface. We observe 3 to 6 packet drops. 

SW-MALI-MOB03#show interfaces Te2/2/5
TenGigabitEthernet2/2/5 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 5c83.8ff5.f874 (bia 5c83.8ff5.f874)
Description: *** Connected to SLT_Corporate backup ***
MTU 9170 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 18/255, rxload 11/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseT
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:13, output never, output hang never
Last clearing of "show interface" counters 00:03:42
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 46962000 bits/sec, 15662 packets/sec
5 minute output rate 71299000 bits/sec, 16974 packets/sec
3371915 packets input, 1161585669 bytes, 0 no buffer
Received 543 broadcasts (267 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
3626422 packets output, 1800277657 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

[Joseph W. Doherty]

No drops recorded on the interface.

As to your few ping drops, pinging a Cisco device, and seeing drops, doesn't necessarily indicate transit packet drops.  Cisco network devices can be a bit "fussy" dealing with ping requests, to themselves, as in delaying or dropping packets when they are otherwise busy or suspect a DoS attack.

What you want to do is use some form of traffic test tool, end to end, and see if it registers any drops.

Also, BTW, less than 1% drops, is generally considered acceptable (for many network applications), regardless of cause.