cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1026
Views
5
Helpful
8
Replies

3850 priority queue drops

paul amaral
Level 4
Level 4

I have a 3850 that I’m trying to figure out what is going on with queue drops on threshold 1 on the priority queue. I'm using 1P7Q3T with auto QOS on a ten gig interface with a CIR of 2 Gig setup using the bandwidth command. 

 

Below is how the queue buffers are setup, (default)

 

DATA Port:5 GPN:3 AFD:Disabled FlatAFD:Disabled QoSMap:3 HW Queues: 40 - 47

  DrainFast:Disabled PortSoftStart:4 - 1872

----------------------------------------------------------

   DTS  Hardmax  Softmax   PortSMin  GlblSMin  PortStEnd

  ----- --------  --------  --------  --------  ---------

 0   1  7   187   9   187   5     0   0     0   5  2496

 1   1  4     0  10   320   8   320   5   125   5  2496

 2   1  4     0  11   500   8   320   5   125   5  2496

 3   1  4     0  11   500   8   320   5   125   5  2496

 4   1  4     0  11   500   8   320   5   125   5  2496

 5   1  4     0  11   500   8   320   5   125   5  2496

 6   1  4     0  11   500   8   320   5   125   5  2496

 7   1  4     0  12  1248   9   798   6   312   5  2496

 Priority   Shaped/shared   weight  shaping_step  sharpedWeight

 --------   -------------   ------  ------------   -------------

 0      1     Shaped          6375         255           0

 1      7     Shared           125           0           0

 2      7     Shared           125           0           0

 3      7     Shared           125           0           0

 4      7     Shared           250           0           0

 5      7     Shared           250           0           0

 6      7     Shared           125           0           0

 7      7     Shared            50           0           0

 

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1  Weight2 Max_Th2 Min_Th2

   ------- ------- ------- ------- ------- -------  ------- ------- ------

 0       0     298       0       0     333       0       0     374       0

 1       0     255       0       0     285       0       0     320       0

 2       0     398       0       0     445       0       0     500       0

 3       0     398       0       0     445       0       0     500       0

 4       0     398       0       0     445       0       0     500       0

 5       0     398       0       0     445       0       0     500       0

 6       0     398       0       0     445       0       0     500       0

 7       0     994       0       0    1111       0       0    1248       0

 

The problem is I'm getting discards on that interface and looking at the queue stats it appears to be on the priority queue on threshold 1 and I'm not sure why that is. I'm having a hard time figuring out why I'm dropping priority packets on queue0 at threshold 1,

 

DATA Port:5 Enqueue Counters
---------------------------------------------------------------------------------------------
Q Buffers          Enqueue-TH0          Enqueue-TH1          Enqueue-TH2             Qpolicer
  (Count)              (Bytes)              (Bytes)              (Bytes)              (Bytes)
- ------- -------------------- -------------------- -------------------- --------------------
0       0                    0           3040662254              1033265                    0
1       0              1805733             25876676               357248                    0
2       0                    0                    0                    0                    0
3       0                    0                    0            727548867                    0
4       0                    0                    0           1341365282                    0
5       0                    0                    0          35966063126                    0
6       0                    0                    0              2468421                    0
7       0                    0                    0          11016902233                    0
DATA Port:5 Drop Counters
-------------------------------------------------------------------------------------------------------------------------------
Q             Drop-TH0             Drop-TH1             Drop-TH2             SBufDrop              QebDrop         QpolicerDrop
               (Bytes)              (Bytes)              (Bytes)              (Bytes)              (Bytes)              (Bytes)
- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
0                    0              4983541                    0                    0                    0                    0
1                    0                    0                    0                    0                    0                    0
2                    0                    0                    0                    0                    0                    0
3                    0                    0                    0                    0                    0                    0
4                    0                    0                    0                    0                    0                    0
5                    0                    0                    0                    0                    0                    0
6                    0                    0                    0                    0                    0                    0
7                    0                    0                    0                    0                    0                    0
AGGR-CAT3K-FRV-MA-AGGR-SW#

 

 

----------------------------------------------------------
   DTS  Hardmax  Softmax   PortSMin  GlblSMin  PortStEnd
  ----- --------  --------  --------  --------  ---------
 0   1  7   187   9   187   5     0   0     0   5  2496
 1   1  4     0  10   320   8   320   5   125   5  2496
 2   1  4     0  11   500   8   320   5   125   5  2496
 3   1  4     0  11   500   8   320   5   125   5  2496
 4   1  4     0  11   500   8   320   5   125   5  2496
 5   1  4     0  11   500   8   320   5   125   5  2496
 6   1  4     0  11   500   8   320   5   125   5  2496
 7   1  4     0  12  1248   9   798   6   312   5  2496
 Priority   Shaped/shared   weight  shaping_step  sharpedWeight
 --------   -------------   ------  ------------   -------------
 0      1     Shaped          6375         255           0
 1      7     Shared           125           0           0
 2      7     Shared           125           0           0
 3      7     Shared           125           0           0
 4      7     Shared           250           0           0
 5      7     Shared           250           0           0
 6      7     Shared           125           0           0
 7      7     Shared            50           0           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1  Weight2 Max_Th2 Min_Th2
   ------- ------- ------- ------- ------- -------  ------- ------- ------
 0       0     298       0       0     333       0       0     374       0
 1       0     255       0       0     285       0       0     320       0
 2       0     398       0       0     445       0       0     500       0
 3       0     398       0       0     445       0       0     500       0
 4       0     398       0       0     445       0       0     500       0
 5       0     398       0       0     445       0       0     500       0
 6       0     398       0       0     445       0       0     500       0
 7       0     994       0       0    1111       0       0    1248       0

 

 

Threshold 0 is set to 298  TH 1 has 333 and TH 2 374 but it looks like when a packet is over threshold one it gets dropped. Not sure how a packet will be determined to go to TH1 or TH2 and how I can fix this issue. Right now I see discards on the output interface and I'm afraid I'm dropping priority packets like voice.

 

TenGigabitEthernet1/0/3 is up, line protocol is up (connected)
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4702711

 

 

Could someone explain to me how the thresholds work on the 3850, why I don’t seem to be using threshold 0 or 2 and what I can do to prevent these drops.

 

TIA, Paul

 

 

 

 

8 Replies 8

balaji.bandi
Hall of Fame
Hall of Fame

worth adding the command  and clear the counter and observe,

 

qos queue-softmax-multiplier 1200

 

you can look at the troubleshooting document for more information.

 

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

paul amaral
Level 4
Level 4

This will only increase the queue for not priority queues, from above the issue im having is drops on the priority queue on TH1, which i read is the default. Using your suggested command will not help with the priority queue drops

noted that the suggested one not worked. let me re-visit the full Q and update my comments, meantime try other suggestions in the post.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Joseph W. Doherty
Hall of Fame
Hall of Fame

I'm no expert on the 3850, but I do wonder about "shared" in regards to Q0.

I also wonder if PQ on a 3850 is more like a router's LLQ, with an implicit policer rather than the earlier 3750s whose PQ was "pure" PQ.

What's the actual auto-QoS that's being used?

Hi, Joseph, Q0 says shaped not shared. Having a hard time trying to figure out how Cisco sets up these Queues, best info I go was from https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html but its not for IOS 16.x

 

sh policy-map AutoQos-4.0-Output-Policy
  Policy Map AutoQos-4.0-Output-Policy
    Class AutoQos-4.0-Output-Priority-Queue
      priority level 1 20 (%)
    Class AutoQos-4.0-Output-Control-Mgmt-Queue
      bandwidth remaining 10 (%)
      queue-limit dscp cs2 percent 80
      queue-limit dscp cs3 percent 90
      queue-limit dscp cs6 percent 100
      queue-limit dscp cs7 percent 100
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Multimedia-Conf-Queue
      bandwidth remaining 10 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Trans-Data-Queue
      bandwidth remaining 10 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Bulk-Data-Queue
      bandwidth remaining 5 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Scavenger-Queue
      bandwidth remaining 5 (%)
      queue-buffers ratio 10
    Class AutoQos-4.0-Output-Multimedia-Strm-Queue
      bandwidth remaining 10 (%)
      queue-buffers ratio 10
    Class class-default
      bandwidth remaining 25 (%)
      queue-buffers ratio 25

I changed some of the bandwidth remaining as I found the default 30% for the priority queue too high. I also noticed that the shared queues have by default low default buffers which I can adjust with the queue-softmax-multiplier command and the queue ration command inside the class map, just not sure why I'm getting dropped packets on TH1 for the priority Q which is the default TH from what I read. Also not sure if there a way to adjust the priority TH1 and if  its even worth it as i will introduce more delay but i dont feel comfortable with the drops. Could the auto qos Policy priority queue just be matching too many packets? for the TH1 level, usually on priority queues I have just EF when I create a policy manually. 

 

DATA Port:5 Drop Counters
-------------------------------------------------------------------------------------------------------------------------------
Q             Drop-TH0             Drop-TH1             Drop-TH2             SBufDrop              QebDrop         QpolicerDrop
               (Bytes)              (Bytes)              (Bytes)              (Bytes)              (Bytes)              (Bytes)
- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
0                    0                18216                    0                    0                    0                    0

 Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 18216

TIA, Paul

 

 

 

I actually was able to increase the priority Thresholds by dropping the queue-buffers ratio on the scavenger class to 5. 

 

DATA Port:5 GPN:3 AFD:Disabled FlatAFD:Disabled QoSMap:3 HW Queues: 40 - 47
  DrainFast:Disabled PortSoftStart:4 - 1872
----------------------------------------------------------
   DTS  Hardmax  Softmax   PortSMin  GlblSMin  PortStEnd
  ----- --------  --------  --------  --------  ---------
 0   1  7   250   8   250   5     0   0     0   5  2496
 1   1  4     0  10   320   8   320   5   125   5  2496
 2   1  4     0  11   500   8   320   5   125   5  2496
 3   1  4     0  11   500   8   320   5   125   5  2496
 4   1  4     0  11   500   8   320   5   125   5  2496
 5   1  4     0  12   248   9   158   6    62   5  2496
 6   1  4     0  11   500   8   320   5   125   5  2496
 7   1  4     0  13  1248  10   798   7   312   5  2496
 Priority   Shaped/shared   weight  shaping_step  sharpedWeight
 --------   -------------   ------  ------------   -------------
 0      1     Shaped          6375         255           0
 1      7     Shared           125           0           0
 2      7     Shared           125           0           0
 3      7     Shared           125           0           0
 4      7     Shared           250           0           0
 5      7     Shared           250           0           0
 6      7     Shared           125           0           0
 7      7     Shared            50           0           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1  Weight2 Max_Th2 Min_Th2
   ------- ------- ------- ------- ------- -------  ------- ------- ------
 0       0     398       0       0     445       0       0     500       0
 1       0     255       0       0     285       0       0     320       0
 2       0     398       0       0     445       0       0     500       0
 3       0     398       0       0     445       0       0     500       0
 4       0     398       0       0     445       0       0     500       0
 5       0     197       0       0     220       0       0     248       0
 6       0     398       0       0     445       0       0     500       0
 7       0     994       0       0    1111       0       0    1248       0
AGGR-CAT3K-FRV-MA-AGGR-SW#

"Hi, Joseph, Q0 says shaped not shared."

Oops, indeed.  I meant to write shaped (as all the other queues are shared).

Looking at your auto-QoS policy-map, the "shaping", likely comes from the bandwidth allocation.  Unclear (to me) whether excess is actually shaped or policed.  (I would expect the latter to avoid PQ adding latency.)

Anyway, recently there was another 3850 poster having PQ drops issues, and I thought the queue limits were too small.  Had trouble finding a way to increase them, because as you noted, qos queue-softmax-multiplier didn't seem to make a difference on PQ.

That person was using both level 1 and level 2 PQ, which also has unclear (as in, IMO, not being well documented) implications.  From reading the documentation using just one PQ and/or one PQ w/o a specified level, might also impact how thresholds are assigned.

As to how packets are assigned to queues, the auto-QoS class-maps, would, hopefully, reveal that.

BTW, my impression was that:

Hardmax  Softmax   PortSMin  GlblSMin  PortStEnd

values might all be adjustable, but dependent on what "knobs" a particular IOS version supports.

You can definitely influence, increase the PQ limits by adjusting the queue-buffers ratio in other non PQ queues. 

Thanks again for your input, I appreciate it as usual. 

 

Paul 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: