Cisco 3650 packet loss, output drops and "no adjacency drops"

[Skjoedt]

Hello

I have a 3650 switch running 16.12.4, which has intermittent packet/connection drops, sometimes really bad drops where video will cut out or audio will drop, when having teams calls or streaming.

I have tried to research the issue and I can see some of my symptoms are:

- High number of output drops on the interface facing my ASA5506X

- High number of "no adjacency" drops, when I do a "show ip traffic"

The number of output drops doesn't move much, however the "no adjacency" drops keeps increasing, with several thousand each minute...

I saw some of the same symptoms in this discussion:
https://community.cisco.com/t5/switching/packet-loss-across-catalyst/m-p/3402007
However I do not have the insight, to know if I am experiencing the same issues?
My ping to Google-DNS is about 15-20% packet loss and from client to switch/gateway is upwards of 20-30% packet loss...

Although the bug is very old, I also found this bug:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva89409/

Would it make sense to try and downgrade to 16.9.6?
- I am mostly running 16.12.4 because of my CCIE Wireless studies...

I hope some will have the insight to help with this issue.

Best Regards

Frederik

[Reza Sharifi]

Hi,

If the issue is only with the port that connects to the firewall, can you try using a different port? Is the cable fine? Is the firewall working as expected? If none of these is the issue then loading a different software version may be helpful.

HTH

[Skjoedt]

Hello Reza

Thank you for replying to my post.

The issues also happens when I ping my default gateway, which is an SVI on the switch.

Both from wired and wireless.
So the firewall is not at fault, since that traffic never reaches the firewall, but still experiences drops/issues.

I have tried downgrading to 16.9.6, but the fans are always at max speed now for same reason, which I cannot live with noise wise...

So I might need to upgrade back to 16.12.4 again...

Best Regards

Frederik

[andrew.butterworth]

Please the describe the topology from a L2 & L3 perspective.  You appear to have a single C3650 switch and an ASA5506-X firewall.  You say that your host's default gateway is an SVI interface on the switch, but you fail to explain how this then routes to the ASA5506-X - is it the same VLAN - I hope not?

I am not sure why an Auto-MGig port being forced at 1Gbps would fix this as once its negotiated the speed/duplex it should stick at this until there is a physical disconnection?

Packet drops due to no adjacency would be due to a lack of an ARP entry for the next-hop?

[Skjoedt]

Hello Andrew

Thank you for replying to my Post.

The Client is on VLAN10 172.16.0.0/24 and there is a Mgmt. subnet/Transit Subnet on VLAN100 172.21.0.0/24 between the ASA5506X and the 3650 switch.
The 3650 Switch is the only switch in the topology (Home Lab/Network).

Traffic is routed from the client to the SVI in VLAN10, then via default route to the ASA5506X on VLAN100, then to the internet.

I think that the reason why forcing 1G operation solved this issue is because of bad drivers on my computers mGig NIC, causing an unstable connection.

The missing ARP-entry cause, could that also be because the host is not responding?
Like if my computers NIC was non-responsive?

Best Regards
Frederik

[andrew.butterworth]

I suspect you are correct with regards to the mGig adapter and drivers if forcing it solves the issue.

There are similar posts elsewhere that mention Realtek mGig adapter issues.  Might be worth sticking a 1Gbps PCIe adapter in the PC and disconnecting the mGig one and seeing how that performs?

[Skjoedt]

Hello Andrew

And that's exactly the NIC i have in my Motherboard, the Realtek mGig adapter, luckily my motherboard also had an Intel I211 NIC, which is just 1G.

I am now using that adapter and have had no instability issues since then...

I used WinMTR on my intel adapter to transmit 1.8 to 1.9 million ping packets over 48 hours with a frequency of 100ms to my default gateway and i had 0 packets lost, while using the same mGig port i used for my Realtek Adapter.

So for now i will use my Intel Adapter until drivers for the Realtek adapter improves.

Basically i did what you just suggested

[MHM Cisco World]

i think QoS of SW is issue here,

lower queue depth make new packet drop.

are this sw in aggregate layer and connect to many underlay access sw?

[Skjoedt]

Hello MHM

Thank you for replying to my post.

The switch is not in the aggregation layer, the switch is the only switch I have operational, so the switch is not connected to other switches.
The switch is only connected to AP's, WLC's, Firewall, Servers, Clients etc.

I read about QoS Queue Depth in the other discussion, but I have no idea on how to fix that... I would need some help to do that.

Best Regards

Frederik

[Skjoedt]

The problem seems to be with mGig.

I had a CCIE R&S colleague take a look at it and after we fixed my switchport to 1G instead of 2.5G mGig, the packet loss disappeared.
If I switched back to 2.5G mGig, the packet loss re-appeared.

Currently I am running 1G on the same mGig port in switch, and I have had no packet loss for several days...

[MHM Cisco World]

...

[Georg Pauwen]

Hello,

if your switch supports the command, try and configure:

qos queue-softmax-multiplier 1200

[Skjoedt]

Hello Georg

Thank you for replying.

I will look into that!
What does the command do?

Best Regards
Frederik

[Georg Pauwen]

Hello,

when you issue the command:

show platform qos queue config interface_x

the output gives you a 'Softmax' column, which basically is the amount of software queue buffers assigned to that interface. The command 'qos queue-softmax-multiplier 1200' increases these values.

You can alos calculate queue and buffers values manually, but the command 'qos queue-softmax-multiplier 1200' is sort of a general value that often decreases or eliminates drops.

[Skjoedt]

Hello Georg

Ohh ok, thank you very much for the clarification, i will let you know if i get around to implement the command and if it eliminates the issues.

Best Regards

Frederik