Similar output drops between Port Channel and Trunk port - 3850-12X48U

[deehusky79]

Hi, anyone ever experiencing this:

SW01#sh int Te1/1/6 --> trunk port to another switch
TenGigabitEthernet1/1/6 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is 0041.d244.c93a (bia 0041.d244.c93a)
  Description: DCI link to CORE-SW01 Te1/1/6
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 8/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:37, output never, output hang never
  Last clearing of "show interface" counters 2w3d
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 8959544
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 34416000 bits/sec, 4897 packets/sec
  5 minute output rate 13972000 bits/sec, 3220 packets/sec
     6271837889 packets input, 5872913878346 bytes, 0 no buffer
     Received 173115 broadcasts (28352 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 28352 multicast, 0 pause input
     0 input packets with dribble condition detected
     6052394670 packets output, 5429935767926 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

SW01#sh int Te1/0/47 --> one of the port channel members to the backup switch
TenGigabitEthernet1/0/47 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is 0041.d244.c92f (bia 0041.d244.c92f)
  Description: to SW02 Te1/0/47
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, media type is 100/1000/2.5G/5G/10GBaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:03, output never, output hang never
  Last clearing of "show interface" counters 2w3d
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 8959544
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 27569000 bits/sec, 4061 packets/sec
  5 minute output rate 21868000 bits/sec, 5595 packets/sec
     2426764983 packets input, 1616841377555 bytes, 0 no buffer
     Received 3947266 broadcasts (135007 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 135007 multicast, 0 pause input
     0 input packets with dribble condition detected
     3453623509 packets output, 2023495432878 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Te1/0/47(P) Te1/0/48(P)

I also would like to stop the drops by applying:

  Policy Map TEST_BUFFER
    Class class-default
      bandwidth 100 (%)
      queue-buffers ratio 100

But since the auto QoS has been used on another port (and I can't remove it), I need to adjust my solution. Can anyone help me how to make adjustment Policy Map TEST_BUFFER so I can stop the output drops on Te1/1/6.

  Policy Map AutoQos-4.0-Output-Policy
    Class AutoQos-4.0-Output-Priority-Queue
      priority level 1 30 (%)
    Class AutoQos-4.0-Output-Control-Mgmt-Queue
      bandwidth remaining 10 (%)
      queue-limit dscp cs2 percent 80
      queue-limit dscp cs3 percent 90
      queue-limit dscp cs6 percent 100
      queue-limit dscp cs7 percent 100
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Multimedia-Conf-Queue
      bandwidth remaining 10 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Trans-Data-Queue
      bandwidth remaining 10 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Bulk-Data-Queue
      bandwidth remaining 4 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Scavenger-Queue
      bandwidth remaining 1 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Multimedia-Strm-Queue
      bandwidth remaining 10 (%)
      queue-buffers ratio 10
    Class class-default
      bandwidth remaining 25 (%)
      queue-buffers ratio 25

  Policy Map AutoQos-4.0-Trust-Cos-Input-Policy
    Class class-default
      set cos cos table AutoQos-4.0-Trust-Cos-Table

Thanks heaps for your time to assist me on this.

[Francesco Molino]

Hi

Drops on the interface aren't all bad. It can be by design if the uper layer is 10G and downstream is 1G.
You can build your own policy-map and apply it to a specific interface. You'll need to remove the auto qos in that specific interfaces before applying your policy-map.

But before creating your policy-map, i would recommend to create your business class-map like voice, video,... and default. Assign the right percentage of allowed bandwidth for each classes otherwise it can be worse than today.

To troubleshooting your drops, you can paste the output of show interface te1/1/6 switching and show interface te1/1/6 stats...

[deehusky79]

Hi Francesco, thanks for the response.

So are you saying this will work: remove the auto QoS (say from port A) and put a customised port on Te1/1/6 and re-apply the auto QoS on port A?

Do you know why the drops value of Te1/1/6 exactly match with Te1/0/47?

Here's the output of show command that you suggested:

SW01#sh int Te1/1/6 switch
Name: Te1/1/6
Switchport: Enabled
Administrative Mode: trunk
Operational Mode: trunk
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: Off
Access Mode VLAN: 1 (default)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: 67,725,830
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Appliance trust: none

=====

SW01#sh int Te1/1/6 stats
TenGigabitEthernet1/1/6
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 28438 13901507 0 0
Route cache 0 0 2329711 161243676
Total 28438 13901507 2329711 161243676

Thank You!

[Francesco Molino]

If you remove auto qos and apply your policy-map, you can't reapply the auto qos. That's why i said you'll need to define all class-map correctly.

Can you rerun the command by typing the complete word switching: sh int te1/1/6 switching

[deehusky79]

I see. Thanks for clarifying it.

SW01#sh int Te1/1/6 switching
TenGigabitEthernet1/1/6 DCI link to CORE-SW01 Te1/1/6

    Protocol  Spanning Tree
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    2022330  129429120   64230011 4110720704
            Cache misses          0          -          -          -
                    Fast          0          0   64230011 4110720704
               Auton/SSE          0          0          0          0

    Protocol  CDP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     882438  431512188     883880  433984962
            Cache misses          0          -          -          -
                    Fast          0          0     883880  433984962
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0     288252   27325156
            Cache misses          0          -          -          -
                    Fast          0          0     288252   27325156
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.

[Francesco Molino · Accepted solution]

Thanks.

You can configure the test policy-map and increase the buffer. I guess you already looked at this guide but i paste the link in case not:
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html?referring_site=RE&pos=2&page=https://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/6343-queue-drop...

Doing that you'll certainly remove drops.
You need to monitor that interface to see what type of traffic is really passing through to figure out why you have lot of drops.

Do you have voice traffic on your network?
If yes you can begin with your policy-map for few days and then add a class-map for voice traffic as priority level 1 and so forth for all classes you need for each type of traffic.