cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5556
Views
35
Helpful
40
Replies

Cisco 3850 | 16.6.8 | Total output drops

Alan.A
Level 1
Level 1

I have 3850 switch running in version 16.6.8 and few interfaces having packet drops, like connections going to access points.after a bit of research I come up with this configuration. Is this correct? This 3850 switches are use only for end users (wired, Phones and wireless (no servers).

conf t
!
qos queue-softmax-multiplier 1200
!
ip access-list extended ACL_QUEUE1
permit ip any any
!
class-map match-any CMAPQUEUE1
match access-group name ACL_QUEUE1
!
policy-map QUEUE1
class class-default
bandwidth percent 100


To be applied on interfaces going to Access Points and Uplink to Core
interface GigabitEthernetX/0/X
service-policy output QUEUE1

 

This is a sample of interface with AP connected

GigabitEthernet1/0/48 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 00a5.bfdf.0b30 (bia 00a5.bfdf.0b30)
Description: ***AP8 ***
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 8w2d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 20083437
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 2 packets/sec
5 minute output rate 2000 bits/sec, 2 packets/sec
303584396 packets input, 167930756495 bytes, 0 no buffer
Received 5931092 broadcasts (5219146 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5219146 multicast, 0 pause input
0 input packets with dribble condition detected
680183326 packets output, 528204738590 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

40 Replies 40

Nothing wrong testing in a lab first, although unless you have lots of expensive traffic generators, often very difficult to replicate the production environment.  I.e. unfortunately, QoS changes actual impact might only been seen in production, especial when issue is at its worse.

Something I haven't mentioned, but possible should, what I do is let management/users know (and agree) there will be a QoS change made at some specific time (during production hours) and immediately report any (worse) performance issues.  Further, I intensely monitor device stats for the first five minutes, and continue to monitor device stats for the next 10 to 25 minutes.  If new (worse) performance issues appear via stats, or via user reports, I'm also ready to do an immediate roll-back.

BTW, much of what @MHM Cisco World and I have suggested, is pretty much also in Cisco's TechNote, including a single queue recommendation configuration, see: example 2 .

Also in TechNote's Case Study: Output Drops we find before pushing queue-softmax-multiplier to 1200:

The drops reduced from 497000 to 385064 for the same traffic burst. Yet, there are still drops. After that, configure qos queue-softmax-multiplier 1200 global config command.

and after setting 1200:

The Softmax for queue-0 can go up to 10,000 buffers and as a result, the drops are Zero.

I.e. what we've suggested, is also suggested by Cisco.

Understand, what we and Cisco suggested might not totally mitigate your drops, and there are other possible side effects, but with Cisco recommending the same, it's not some "wild recommendation" from two people on some support forum (in case you need to justify why to make these particular changes).  ; )

now let explain all steps done 
before do any thing 

----------------------------------------------------------
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 5 120 2 480 6 320 0 0 4 1440
1 1 4 0 6 720 3 480 2 180 4 1440

the default Buffer size is 300 buffers (buffers not bytes)
above is default config which contain two Queue 
Hardmax
Queue1 = 40% of buffer size
Queue0= 0
Softmax
Queue1 = 40% of buffer size multiple by 1 
Queue0= (100-40)% of buffer size multiple by 1

NOW 
qos queue-softmax-multiplier 400

----------------------------------------------------------
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 5 120 7 1920 6 320 0 0 5 5760
1 1 4 0 8 2880 3 480 2 180 5 5760

Hardmax
Queue1 = 40% of buffer size
Queue0= 0
Softmax
Queue1 = 40% of buffer size multiple by 4
Queue0= (100-40)% of buffer size multiple by 4

for why Queue-buffer ratio 100% not work need confirm from @Alan.A , please Alan are you apply service policy to interface or you only adjust the max multiplier ??

Alan.A
Level 1
Level 1

This are the configuration that I just applied in the 3850 switch. I will observe in the next few days/weeks.

!
qos queue-softmax-multiplier 1200
!
XXXXXXXXXXXXXXX#sh run | sec policy-map QUEUE1
policy-map QUEUE1
class class-default
priority level 2
queue-buffers ratio 100
XXXXXXXXXXXXXXX#


interface GigabitEthernet1/0/47
description ***AP7 ***
service-policy output QUEUE1
!
interface GigabitEthernet1/0/48
description ***AP8 ***
service-policy output QUEUE1
end


XXXXXXXXXXXXXXX#show platform hardware fed switch active qos queue config interface gigabitEthernet 1/0/48
DATA Port:26 GPN:48 AFD:Disabled QoSMap:3 HW Queues: 208 - 215
DrainFast:Disabled PortSoftStart:4 - 10000
----------------------------------------------------------
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 6 300 8 10000 9 800 0 0 3 10000
1 1 4 0 5 0 5 0 0 0 3 10000
2 1 4 0 5 0 5 0 0 0 3 10000
3 1 4 0 5 0 5 0 0 0 3 10000
4 1 4 0 5 0 5 0 0 0 3 10000
5 1 4 0 5 0 5 0 0 0 3 10000
6 1 4 0 5 0 5 0 0 0 3 10000
7 1 4 0 5 0 5 0 0 0 3 10000
Priority Shaped/shared weight shaping_step
-------- ------------- ------ ------------
0 2 Shared 50 0
1 0 Shared 10000 0
2 0 Shared 10000 0
3 0 Shared 10000 0
4 0 Shared 10000 0
5 0 Shared 10000 0
6 0 Shared 10000 0
7 0 Shared 10000 0

Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
------- ------- ------- ------- ------- ------- ------- ------- ------
0 0 8207 0 0 9173 0 0 10300 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
XXXXXXXXXXXXXXX#

 

 

Guess we'll see if that QoS configuration changes help mitigate your port drops.

Buffers and drop thresholds, much increased.

Again, if congestion is sustained, this won't help much.  If bursting congestion, it may make a huge difference.

BTW, something I forgot to touch upon before, if ports are connecting to APs, and they (or their hosts) cannot sustain gig or better data transfer rates, you might move the drops from your switch's port to the AP.

Any preliminary/early results?

so far no drops in both interface connecting to Access Point but I will observe this until Friday.

That's a huge improvement, correct?

Double check that your other ports drops rate hasn't increased.

10000 softmax buffer, just want to ask you something, from where this extra buffer size?

Search the TechNote, I referenced, for 10000.

Note: This kind of scenario is not possible as other interfaces can also use the buffer, but, this can definitely help to reduce the packet drops to a certain level.

 

 

From same note you share, and I think he missing read above statement.

10000 buffer meaning that all softmax will be use by only this interface. Other interface will face huge drop issue.

Read note second time.

Note: This kind of scenario is not possible as other interfaces can also use the buffer, but, this can definitely help to reduce the packet drops to a certain level.

Laugh - perhaps TechNote needs to be read a third time?

Right above the that note is (which I believe is the object of the note):

The Softmax for queue-0 can go up to 10,000 buffers and as a result, the drops are Zero.

I.e.  I believe the "scenario" is actually obtaining 10,000 buffers.  Which cannot actually happen, or "is not possible as other interfaces can also use the buffer".

Further, that note ends with:

but, this can definitely help to reduce the packet drops to a certain level.

I.e. in reality, the drops are Zero. is unlikely or not guaranteed with this setting.

What the TechNote doesn't have, is anything like, for this Case Study example is "DON'T TRY THIS AT HOME".  ; )

At to your statements:

"10000 buffer meaning that all softmax will be use by only this interface. Other interface will face huge drop issue."

I would revise those to be:

10000 buffer meaning that all softmax might be used by such an interface. Other interfaces might have increased drops.

Assuming my revised statement are correct, often the only way to know for sure the impact, good or bad, is to try a change.

PS:

BTW, my intent is not to totally discount @MHM Cisco World 's concern, as there really is a possibility of these changes creating an overall negative impact.  Even if that happens, negative results, can lead to later positive results (assuming we learn from the negative results).