3850 strange logging

[tianwen.zhao]

Hello everyone,

Three of our 3850 are in stack which version are 16.6.5 

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 16 WS-C3850-12XS 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE
2 32 WS-C3850-24S 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE
3 56 WS-C3850-48T 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE

We just upgrade to 16.6.5 just a few days ago. But some strange logs are show up

Here are some output:

Feb 26 14:16:25.661 CNT: %PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached.
Feb 26 14:17:25.768 CNT: %PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached.
Feb 26 14:18:25.867 CNT: %PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached.
Feb 26 15:32:24.419 CNT: %PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached.
Feb 26 15:33:24.419 CNT: %PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached.

#sho interfaces port-channel 91

Port-channel91 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 08cc.a77f.26a7 (bia 08cc.a77f.26a7)
Description: link to AC3600
MTU 1500 bytes, BW 4000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is
input flow-control is on, output flow-control is unsupported
Members in this channel: Gi3/0/36 Gi3/0/37 Gi3/0/38 Gi3/0/39
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/4 (size/max/drops/flushes); Total output drops: 4597399
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 21478000 bits/sec, 3803 packets/sec
5 minute output rate 26728000 bits/sec, 4746 packets/sec
397559649 packets input, 309361334081 bytes, 0 no buffer
Received 2427277 broadcasts (1102354 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1102354 multicast, 0 pause input
0 input packets with dribble condition detected
423232681 packets output, 310419459111 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Interface Po91 are have mass  output drops.

Interface Po91 is connect to our AC controller ,Some AP are frequently offline

Can someone please assist me to fix this problem??

Thank you 

[Mark Malone]

Hi

i don't see any specific bugs relating to that message in the release notes of your version below , have you ran the show tech yet to see what it provides ? the analyzer can give a bug ID or may point to a hardware issue or capacity issue

have you tried reloading the stack too , does the problem go away after all processes are reset or does it come back into logging  ? it looks as if some threshold has been hit

: MATM dropped learn cache batch messages as queue limit has reached

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-6/release_notes/ol-16-6-3850.html

[tianwen.zhao]

Thank you for your reply.
Before we upgrade the switches We didn't see that messages .
What messages should I see in show tech?
How can I confirm that is a hardware issue or capacity issue ?
I reload the stack several times when I upgrade the switches. The problem showed up after I upgrade to 16.6.5
We upgrade to 16.6.5 in order to solve the bug CSCvc14038
What I concern that is a hardware issue or capacity issue now . But the switches running normally all the time before we upgrade the version

Thank you

[Mark Malone]

Hi
Use the Cisco analyzer tool on there website link below it can find issues in the switch , bugs , performance issues etc
If that doesn't find anything open a TAC case if your upgrading and still seeing the problem , they can either provide the bug id if the analzyser doesn't or they can advise if its a hardware issue

https://www.cisco.com/c/en/us/support/web/tools-catalog.html

https://cway.cisco.com/docs/cisco-cli-analyzer/2.0/About_the_Cisco_CLI_Analyzer.htm

[Leo Laohoo]

Is QoS enabled?

[tianwen.zhao]

Thank you for your reply.
We have configured "auto qos trust "

[Figuig]

What QoS has to do with this issue?

[Georg Pauwen]

Hello,

check what happens when you configure:

qos queue-softmax-multiplier 1200

on the switches, this should at the very least reduce the output errors (and possibly eliminate the log message).

[AdamF1]

Did you ever find a solution to this?

I have the same issue at the same time each day on a switch stack of 5 running 16.9.6. It happens during the PC reboot schedule each morning while the PC's are coming back offline and re-authenticating on the network. I've looked at our uplinks and ports for saturation at that time and there is none..