Cisco 3850 Switch getting message %SPI_FC-3-HIGH_WMARK_REACHED

[AKIL HASAN]

Hi Team,

We have one Cisco 3850 Switch installed at the Customer site and getting the message as below,

Mar 28 10:57:11.578: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#830db5fd318976b0280defe233875463  :10000000+153F71C :10000000+B5E9B0 :10000000+B5EBA4 :10000000+3CC5CC :10000000+3CD0B4 :10000000+39AB00 :10000000+27CDD2C :10000000+7C7814 :10000000+27AC29C
.Mar 28 10:58:31.585: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#830db5fd318976b0280defe233875463  :10000000+153F71C :10000000+B5E9B0 :10000000+B5EBA4 :10000000+3CC5CC :10000000+3CD0B4 :10000000+39AB00 :10000000+27CDD2C :10000000+7C7814 :10000000+27AC29C
 --More--         .Mar 28 10:59:51.586: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#830db5fd318976b0280defe233875463  :10000000+153F71C :10000000+B5E9B0 :10000000+B5EBA4 :10000000+3CC5CC :10000000+3CD0B4 :10000000+39AB00 :10000000+27CDD2C :10000000+7C7814 :10000000+27AC29C

Please suggest the meaning of these messages and suggestion for resolution on the same.

Attaching show version, show logging for this Switch.

Regards

Ashutosh

[Arunagiri Nallusamy]

Hi Ashutosh,

What is the IOS version being used? I believe you are running into Bug:CSCuh78943. Please upgrade the code to 3.3.2 and the fix is available in the latest code.

Thanks,

Arun

[Nick Gilbert]

I am having this problem on a WS-C3850-48P running 03.03.03SE.

2014-08-01 00:00:12 Local7.Error 10.201.1.4 577868: Jul 31 23:00:11.464: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 13 has reached maximum.

2014-08-01 00:00:12 Local7.Error 10.201.1.4 577869: -Traceback= 1#2464e3fb3a547c97c82963c14d9a54f4  :554BA000+1FA421C :554BA000+12A6C08 :554BA000+12A6DFC :554BA000+AB5C44 :554BA000+AAA628 :554BA000+AABAC0 :554BA000+342F27C

Any help would be appreciated.

Thanks

[educruz]

Hi Akilhasan,

The switch is hitting a bug which is currently under investigation. The latter implies there is no official workaround, but my suggestion would be that you consider reloading the switch outside of business hours (considering preventive measures i. e. back the configuration up, save changes, etc.). 

The most stable version and recommended per Cisco is 03.03.03. There is newer 03.06.00 available, just released past June, so you may consider as well upgrading the IOS, of course under a properly carried out risk assessment, and you can roll back if something unexpected occurs. I would suggest doing so only if you have a solid knowledge of the customer's network and business needs.

Hope this helps. 

Kind regards,

- Ed

[Nick Gilbert]

Thanks for the info Ed

The switch is not in production as of yet, it was being used to test ac wireless so I can reload as and when I choose. I will be installing 03.03.06 and seeing if this resolves the problem.

[6bk.simmons]

So I have a 3850 switch running 03.07.04E and we are seeing the exact same error message. %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 11 has reached maximum.
-Traceback= 1#070eb7d9ba6f0bc87b2562bfd825b241 :54E42000+28478A4 :54E42000+184194C :54E42000+1841B44 :54E42000+10D6450 :54E42000+10D6978 :54E42000+11312DC :54E42000+112CA2C :54E42000+4763FFC.  And I too hit the "not enough permissions" to view bug detail on the Cisco bug website. So reaching out to my Cisco SE to see what he can find out. But it looks like they had a fix for it back in 03.03.03 code. But the bug has reappeared and now Cisco is having to update their bug id. So they pulled the old description and fix page. We are going to upgrade our switch to 03.07.05E. The only other option would be to move to the Denali or Everest code train. 

[sameer kumar chug]

@6bk.simmons I am running with same bug on software version 03.06.06. Have you upgraded it on 03.07.05 and is it resolved now ?

[Roman_Shilenko]

A stack of four 3650s running 03.07.05E - same bug:

117817: May 23 05:34:41 PST: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 11 has reached maximum.
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54CEC000+2865014 :54CEC000+185BDDC :54CEC000+185BFD4 :54CEC000+10EED48 :54CEC000+10EF270 :54CEC000+11177FC :54CEC000+11224D8 :54CEC000+1109F24 :54CEC000+110A188 :54CEC000+254FDD0 :54CEC000+2545328

[chitrang]

Hello Akilhasan,

Stack of 4 3850 switches running 03.06.04.E having the same error.

%SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 11 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :546D6000+25DA3DC :546D6000+167C96C :546D6000+167CB64 :546D6000+F67938 :546D6000+F67E20 :546D6000+FACFF0 :546D6000+FAD54C :546D6000+FB1FC0 :546D6000+3F1897C

[MightyFryer]

I can confirm this bug is present in 03.06.09E as well. Moving to 16.x code during scheduled maintenance next week.

[Mohsin Alam]

Upgrade to latest cisco recommended IOS:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/214814-recommended-releases-for-catalyst-9200-9.html

## Make sure to mark post as helpful, If it resolved your issue. ##

[LajpatSharma17601]

i also having same issue, please refer to below logs and advise.

++++++++++++++++++

*Jan 12 06:58:45.896 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:00:05.900 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:01:25.906 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:02:45.907 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:04:05.917 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:05:25.929 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:06:45.941 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:08:05.947 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:09:25.958 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:10:45.967 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:12:05.979 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:13:25.988 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:14:45.996 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:16:05.996 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:17:25.997 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:18:46.006 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:20:06.009 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:21:26.017 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:22:46.026 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:24:06.029 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:25:26.037 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:26:46.050 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:28:06.058 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:29:26.068 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:30:46.078 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:32:06.090 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:33:26.096 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:34:46.097 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:36:06.106 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:37:26.106 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:38:46.108 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:40:06.120 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:41:26.129 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:42:46.137 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:44:06.146 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:45:26.147 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:46:46.146 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:48:06.147 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:49:26.148 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:50:46.158 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:52:06.169 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:53:26.180 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:54:46.188 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:56:06.197 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:57:26.209 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 07:58:46.217 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:00:06.228 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:01:26.238 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:02:46.248 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:04:06.256 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:05:26.256 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:06:46.266 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:08:06.267 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:09:26.277 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:10:46.289 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:12:06.299 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
*Jan 12 08:13:26.309 UTC: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 9 has reached maximum.
-Traceback= 1#b5439af4a89a33e14c7616c14e7451c0 :54B1A000+25DA3DC :54B1A000+167C96C :54B1A000+167CB64 :54B1A000+E263E4 :54B1A000+E26F34 :54B1A000+DE91B0 :54B1A000+3F3529C :54B1A000+12AA364 :54B1A000+3F1897C
wwbntsts04c36#sh ver
wwbntsts04c36#sh version
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.06.04.E RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Sat 13-Feb-16 03:45 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)

wwbntsts04c36 uptime is 2 years, 9 weeks, 40 minutes
Uptime for this control processor is 2 years, 9 weeks, 43 minutes
System returned to ROM by Power Failure
System image file is "flash:packages.conf"
Last reload reason: Power Failure

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipbase
License Type: Permanent
Next reload license Level: Ipbase

cisco WS-C3650-24PS (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FDO2215I1A5
2 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of at webui:.

Base Ethernet MAC Address : 70:0f:6a:55:f3:00
Motherboard Assembly Number : 73-15899-07
Motherboard Serial Number : FDO22152BKJ
Model Revision Number : S0
Motherboard Revision Number : A0
Model Number : WS-C3650-24PS
System Serial Number : FDO2215I1A5

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24PS 03.06.04.E cat3k_caa-universalk9 INSTALL

Configuration register is 0x102

wwbntsts04c36#

[Georg Pauwen]

Hello,

try and upgrade to the recommended release, Gibraltar-16.12.5b MD.

Also, the switch uptime is > 2 years. Rebooting the switch might help...