Output drops in a c3850 with 100 Mbps negotiated ports for 6921 Phones

[Jmartin79]

Hello,

Recently it has gotten my attention that our network with phones that have 100 mbps ports connected to 1 Ge ports of switches like that of a 3850 (16.12.5b)  are experiencing massive output drops like easily reaching 1 - 3 million drops in a day. I thought it was the WAN, but it's barely congested or reaches its max. Other phones like 9971s which come with 1GB ports aren't seeing this.

I tried following the Cisco documentation of putting the phones with automatic port sync and leaving remote port to auto-negotiate in Call Manager to supposedly to prevent packet loss, but drops are still occurring. 

I tried putting it to fixed 100 on both ends and same issue.

Anyone experience this?

here's an output of all ports with 6921 phones:

sh interface | i Total output drops: [1-9]+
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 3384391
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 3223112
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1347049
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 3553816
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 3495173
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1096150
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2262041
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2410591
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 5316889
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 239693
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2626070
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1117232
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1206475
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 3842886

Switch uptime : 1 day, 11 hours, 12 minutes

[balaji.bandi]

Can you post one of the port information full output

show run interface x/x

sh interface x/x controller

show interface x/x _to understand the issue.

show run | i softmax

try

qos queue-softmax-multiplier 1200

[Jmartin79]

Here are the typical configs for switch ports:

sh interface GigabitEthernet2/0/17
description edge_port_ISE
switchport access vlan 2
switchport mode access
switchport voice vlan 3759
device-tracking attach-policy IPDT_POLICY
load-interval 30
authentication periodic
authentication timer reauthenticate 32400
access-session host-mode multi-domain
access-session control-direction in
access-session closed
access-session port-control auto
mab
no snmp trap link-status
dot1x pae authenticator
dot1x timeout quiet-period 5
dot1x timeout tx-period 10
dot1x timeout supp-timeout 3
dot1x max-reauth-req 3
storm-control broadcast level 1.00
storm-control multicast level 1.00
storm-control action shutdown
storm-control action trap
spanning-tree portfast
spanning-tree bpduguard enable
service-policy type control subscriber IDENTITY_POLICY_DOT1X_MAB

sh int g2/0/17

GigabitEthernet2/0/17 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 5006.ab50.5791 (bia 5006.ab50.5791)
Description: edge_port_ISE
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:49, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d17h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1526291
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 68000 bits/sec, 9 packets/sec
30 second output rate 18000 bits/sec, 10 packets/sec
136530 packets input, 53592478 bytes, 0 no buffer
Received 8291 broadcasts (8019 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 8019 multicast, 0 pause input
0 input packets with dribble condition detected
617731 packets output, 359149059 bytes, 0 underruns
Output 22508 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
3917 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

After a few seconds drops already increased mostly by accessing a website or anything burst heavy:

sh int g2/0/17 controller
GigabitEthernet2/0/17 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 5006.ab50.5791 (bia 5006.ab50.5791)
Description: edge_port_ISE
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:50, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d17h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1741013
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 64000 bits/sec, 31 packets/sec
30 second output rate 1434000 bits/sec, 143 packets/sec
143064 packets input, 55146887 bytes, 0 no buffer
Received 8297 broadcasts (8025 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 8025 multicast, 0 pause input
0 input packets with dribble condition detected
651788 packets output, 404248909 bytes, 0 underruns
Output 22728 broadcasts (0 multicasts)
0 output errors, 0 collisions, 0 interface resets
3919 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet2/0/17 Receive
1562888486 Total bytes 401968403 Total bytes
2277987 Unicast frames 1354438 Unicast frames
1486478682 Unicast bytes 398609726 Unicast bytes
616196 Multicast frames 14567 Multicast frames
48923130 Multicast bytes 3026482 Multicast bytes
240637 Broadcast frames 3009 Broadcast frames
27486674 Broadcast bytes 332195 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
570537 Minimum size frames 407019 Minimum size frames
784778 65 to 127 byte frames 118611 65 to 127 byte frames
649591 128 to 255 byte frames 570562 128 to 255 byte frames
168742 256 to 511 byte frames 81173 256 to 511 byte frames
69181 512 to 1023 byte frames 48787 512 to 1023 byte frames
891991 1024 to 1518 byte frames 145862 1024 to 1518 byte frames
0 1519 to 2047 byte frames 0 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
5938813 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

Phone info via cdp:

Platform: Cisco IP Phone 6921, Capabilities: Host Phone Two-port Mac Relay
Interface: GigabitEthernet2/0/17, Port ID (outgoing port): Port 1
Holdtime : 145 sec
Second Port Status: Up

Version :
SIP 9.4.1.3.SR3

advertisement version: 2
Duplex: full
Power drawn: 6.000 Watts
Power request id: 17187, Power management id: 3
Power request levels are:6000 0 0 0 0

no softmax setting configured, it's not an available command it appears on 16.12.5b.

Since it's copper cables, I ran the TDR command  and all four pairs are reporting normal.

Thanks for the info.

[Jmartin79]

Update:

After doing some more research the command to set speed auto to negotiate at 100 for these ports and seems to do the trick. I even removed prior auto-negotiate configs in call manager.

I had only tried putting it fixed at 100 and still saw the errors which is why I didn't try speed auto 100. Weird. 

Edit: 

NVM...drops returned.

I also just realized the softmax command is a global not port specific. Let's see what happens.

[Jmartin79]

I realized the softmax setting is a global config, not an interface config.

[Joseph W. Doherty]

Are there only phones on these ports or phone and downstream PC?

[Jmartin79]

PCs connected to PC port of the phones. So something doesn't like 1 GbE NICs to these phones :(.

[Joseph W. Doherty]

More likely, sending gig, or better, transmission rate to these ports, and with a 100 Mbps inteface, their buffers will fill rapidly, causing drops.

Your already using the softmax 1200 setting?

If so, unsure 3850s support, but on the prior 3560/3750 could change common pool and interface pool reserve buffers, allowing more buffer space for an "busy" port.  Offen doing this would mitigate port output drops, but doing so works best for one or few "busy" ports.

[Elliot Dierksen]

You could also be having a problem with the internal switch in the phone. Those are SUPPOSED to be able to deal with mixed speeds between the network uplink and the PC port, but my experience with that is has been less than stellar. What I do is enable 'automatic port synchronization' in the product specific configuration layout of the phone in question. I actually prefer to enable that in enterprise phone configuration. Perhaps that will help. Besides, it doesn't do the PC behind the phone any good to have a gigabit connection to the phone if the phones uplink is 100Mb anyway.

[Jmartin79]

Yes, I thought so too about the auto port sync configuration in Call Manager, but it didn't do anything. So far what I've done is created a QoS profile like here c3850 QoS   with the soft-queue max command from above.

I do not see any more drops, but the queue is shared among all ports so I haven't activated on many ports yet.