high interface discards between 2 sites

[BhavMistry]

We have high packet discards between 2 sites

primary site has 2 x 10gig sfp's and other side is connected via x2 1gig

both sites have a port channel with the exact same config

speed negotiated at 2gig

interface configuration is the same on both sides.

gi1/1/2 and gi2/1/2 as a etherchannel / port channel

tengi3/1/2 and tengi4/1/2 as etherchannel / port channel

Site 2 is where the high dicards are happening

Site 1

TenGigabitEthernet3/1/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:12, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

TenGigabitEthernet4/1/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Site 2

GigabitEthernet2/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:06, output 00:00:15, output hang never
Last clearing of "show interface" counters 25w0d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 479274732293

GigabitEthernet1/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:05, output 00:00:03, output hang never
Last clearing of "show interface" counters 25w0d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 9035358488

any help would be appreciated

TIA

[Georg Pauwen]

Hello,

not sure what your topology looks like: two 10G interfaces connected to two 1G interfaces ? Output drops usually means there are too many packets trying to get out over a congested line. The obvious solution would be to change the 1 to 10G interfaces...

[Joseph W. Doherty]

"not sure what your topology looks like: two 10G interfaces connected to two 1G interfaces ?"

@Georg Pauwen I believe/suspect what OP might have is 10g SFP+ on one side, using gig SFPs, communicating with other side's gig SFPs.  I.e. gig to gig.

[Flavio Miranda]

Hi

  As I can see, interface statistic is there for a while:

Last clearing of "show interface" counters 25w0d

Try to clear the counter and monitoring. This problem may had gone already and you are looking for ghost.

[Joseph W. Doherty]

Ditto with @Flavio Miranda, i.e worthwhile clearing interface stats and to further see what's happening.

Also, as OP didn't post transmitted packet counts, we don't know drop count percentage; often more important than overall drop count.

Lastly, what's the platform being used showing the non-zero drop counts.

[BhavMistry]

GigabitEthernet1/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 57/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:04, output hang never
Last clearing of "show interface" counters 25w3d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 9145934724
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 223896000 bits/sec, 20165 packets/sec
5 minute output rate 4008000 bits/sec, 4451 packets/sec
74843871132 packets input, 83959987679573 bytes, 0 no buffer
Received 128589586 broadcasts (37599864 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 37599864 multicast, 0 pause input
0 input packets with dribble condition detected
47264591474 packets output, 41973291999727 bytes, 0 underruns
Output 24594809 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

GigabitEthernet2/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:10, output hang never
Last clearing of "show interface" counters 25w3d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 479459697876
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 18401000 bits/sec, 1750 packets/sec
5 minute output rate 1733000 bits/sec, 925 packets/sec
105488892486 packets input, 102633295409587 bytes, 0 no buffer
Received 30767319 broadcasts (14687301 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 14687301 multicast, 0 pause input
0 input packets with dribble condition detected
112736040643 packets output, 145703370584450 bytes, 0 underruns
Output 14653321 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[balaji.bandi]

post complete output of interface to look more information.

what model of the switch or device both the side, post the config also help us.

we see only drops on one side.

Couple of suggestions:

1. any Qos Config ?

2. set the speed config on the port-channel (is that fix the issue)

3. Once you clear the counter, how fast do they increase?

4. are you seeing any performance issue?

5. what LB method is used for Port-channel?

[BhavMistry]

Hi Balaji,

what is the best way to check load balancing for port-channel?

please see interface stats below:

GigabitEthernet1/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 57/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:04, output hang never
Last clearing of "show interface" counters 25w3d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 9145934724
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 223896000 bits/sec, 20165 packets/sec
5 minute output rate 4008000 bits/sec, 4451 packets/sec
74843871132 packets input, 83959987679573 bytes, 0 no buffer
Received 128589586 broadcasts (37599864 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 37599864 multicast, 0 pause input
0 input packets with dribble condition detected
47264591474 packets output, 41973291999727 bytes, 0 underruns
Output 24594809 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

GigabitEthernet2/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:10, output hang never
Last clearing of "show interface" counters 25w3d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 479459697876
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 18401000 bits/sec, 1750 packets/sec
5 minute output rate 1733000 bits/sec, 925 packets/sec
105488892486 packets input, 102633295409587 bytes, 0 no buffer
Received 30767319 broadcasts (14687301 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 14687301 multicast, 0 pause input
0 input packets with dribble condition detected
112736040643 packets output, 145703370584450 bytes, 0 underruns
Output 14653321 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

i have not cleared the counters as of yet but since i posted the count number has gone up.

interface configuration as follows:

interface GigabitEthernet1/1/2
switchport trunk native vlan 110
switchport trunk allowed vlan 110-112,700
switchport mode trunk
channel-group 18 mode active

interface GigabitEthernet2/1/2
switchport trunk native vlan 110
switchport trunk allowed vlan 110-112,700
switchport mode trunk
channel-group 18 mode active

[BhavMistry]

we have this as a global configuration for port-channel load balancing:

port-channel load-balance src-dst-ip

[balaji.bandi]

You can find the PO load-balance method here - change according to your setup :

https://www.cisco.com/c/en/us/support/docs/lan-switching/etherchannel/12023-4.html

Site 1 where we have no drops we are using WS-C3850-48P stacked with 5 switches, IOS version: 03.03.05SE

Site 2 where we do have drops we are using WS-C3850-24T stacked with 2 switches, IOS version: 16.12.05b

can you post other side interface and config output ? is this directly connectd  ? how is your network conencted ?

some troubleshoot tips for output drops 

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

[BhavMistry]

Connection between sites:

SITE 1: Cisco Core > BGP Router > Site 2 >BGP Router > Core
i cleared the counters on Site 2 and they are incamenting after the clear.

Site: 1 iinterface details

TenGigabitEthernet3/1/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 5/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:09, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2720000 bits/sec, 542 packets/sec
5 minute output rate 19988000 bits/sec, 3095 packets/sec
547117241567 packets input, 730086363340482 bytes, 0 no buffer
Received 183055232 broadcasts (61775574 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 61775574 multicast, 0 pause input
0 input packets with dribble condition detected
507568859409 packets output, 520451805578843 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

TenGigabitEthernet4/1/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6348000 bits/sec, 2139 packets/sec
5 minute output rate 8895000 bits/sec, 1100 packets/sec
269921943669 packets input, 269135499567398 bytes, 0 no buffer
Received 312646374 broadcasts (69360964 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 69360964 multicast, 0 pause input
0 input packets with dribble condition detected
372909172525 packets output, 415549556763484 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

interface TenGigabitEthernet3/1/2
switchport trunk native vlan 110
switchport trunk allowed vlan 110-112,700
switchport mode trunk
channel-group 18 mode active
end

interface TenGigabitEthernet4/1/2
switchport trunk native vlan 110
switchport trunk allowed vlan 110-112,700
switchport mode trunk
channel-group 18 mode active
end

[Joseph W. Doherty]

"i cleared the counters on Site 2 and they are incamenting after the clear."

It would be helpful if you would post site 2's interface stats, i.e. the interfaces showing incremental drops.

[BhavMistry]

Site 2 interfaces drops

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Gi1/0/22 0 0 0 0 0 0
Gi1/0/23 0 0 0 0 0 0
Gi1/0/24 0 0 0 0 0 0
Gi1/1/1 0 0 0 0 0 0
Gi1/1/2 0 0 0 0 0 5071481
Te1/1/3 0 0 0 0 0 0

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Gi2/0/16 0 0 0 0 0 0
Gi2/0/17 0 0 0 0 0 0
Gi2/0/18 0 0 0 0 0 0
Gi2/0/19 0 0 0 0 0 0
Gi2/0/20 0 0 0 0 0 0
Gi2/0/21 0 0 0 0 0 0
Gi2/0/22 0 0 0 0 0 0
Gi2/0/23 0 0 0 0 0 0
Gi2/0/24 0 0 0 0 0 0
Gi2/1/1 0 0 0 0 0 0
Gi2/1/2 0 0 0 0 0 9928434

[BhavMistry]

GigabitEthernet2/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 9/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:05, output 00:00:20, output hang never
Last clearing of "show interface" counters 02:02:59
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 9928434
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 37986000 bits/sec, 4511 packets/sec
5 minute output rate 1896000 bits/sec, 1232 packets/sec
29296383 packets input, 28302841147 bytes, 0 no buffer
Received 3894 broadcasts (3412 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3412 multicast, 0 pause input
0 input packets with dribble condition detected
7359702 packets output, 5148524011 bytes, 0 underruns
Output 386 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

GigabitEthernet1/1/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet,
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 4/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:11, output 00:00:07, output hang never
Last clearing of "show interface" counters 02:02:29
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 5215855
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5587000 bits/sec, 1188 packets/sec
5 minute output rate 18858000 bits/sec, 2938 packets/sec
12359564 packets input, 13209262298 bytes, 0 no buffer
Received 28099 broadcasts (16419 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 16419 multicast, 0 pause input
0 input packets with dribble condition detected
14831131 packets output, 6789082926 bytes, 0 underruns
Output 30505 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[balaji.bandi]

ok now Can you post show the router side interface config and show interface output, the one getting output errors and the one not getting

to make it clear, Site 1 switch and Router output  (1 attachement) and Site 2 switch and Router output  (2nd attachement)

so we can ready easily also what Code they running each device.