SVI goes down on 3850s for 5 to 10 minutes at a time

[stonent01]

I have a 3850 that has a 2 x 10G fiber port channel going directly to our 6880x core switch that's a few feet away.

Randomly for 5 or 10 minutes at a time, the management SVI on that 3850 switch stops pinging, but the switch otherwise remains fully functional, and traffic on the other VLANs continues to flow.

When that happens, 2 other 3850 switches that are 2 x 1G port channeled to it also stop pinging on their SVI.  The SVIs are on the same VLAN which 95% of the SVIs are on at our facility.

The logs on all 4 of the switches involved here show nothing at all during this time other than a login entry for me.

Is there anything you can think of that would just cause the traffic on this VLAN to just just stop and come back like this?  It has to be either the 6880x blocking the traffic, or the 3850 blocking the traffic. Other switches running off the same 6880x do not have this issue and the SVI is on the same VLAN as the ones with the problem.

This is not generating any production issues but it is generating alerts from our monitoring software.  It's been doing this for about 6 months now, and none of the switches in question have had any changes to their IOS version or really much of anything at all.  

The 3850 is running Cisco IOS XE Software, Version 16.12.05b, and the core is running Cisco IOS Software, c6880x Software (c6880x-ADVENTERPRISEK9-M), Version 15.5(1)SY8, RELEASE SOFTWARE (fc3).

Thanks for any ideas..

[balaji.bandi]

Can you post the port-channel config and show spanning vlan X from both the switches ?

when you not able to ping

show IP interface brief (do you see the vlan of SVI up ?) or going down ?

from what IP you trying to ping, same subnet ?

[stonent01]

I can get some of that information later but on the question of where is the ping coming from, I've logged into the core switch that it's directly connected to and it won't ping.  The core has lots of SVIs since it's doing routing, so I have also tried from other switches with just a single SVI on that VLAN and no ping there either (most of them are connected directly to the core as well).
I did one time run back there when it went down with my laptop to console in and if I recall, the serial console seemed frozen but after a minute started responding rather slowly but the switch had started pinging after that.

I'm thinking now I should probably set up an SVI on another VLAN and see if that stops pinging during that time and also leave a PC in that cabinet wired to the console port that I can remote into quickly when the e-mail comes through and maybe have a script running with a continuous ping with timestamps dumping to a text file.

[balaji.bandi]

CORE Switch SVI pings ? Only issue Cat 3850 switch ?

Do you have any NMS that you monitor device health, so check any CPU usage?

also, run EEM Script to find out any issues.

[MHM Cisco World]

friend the Ping to management SVI failed because the CPU of SW is high utilize or the Queue is full so the ping either timeout or drop. 
other SVI no effect because the SW have two part to forward frames 
1- HW (any traffic pass through SW) 
2- SW "CPU" any traffic toward the SW or in some case if HW can not forward frame it send to CPU. 

so here all issue come from CPU or Queue to CPU. 

[Leo Laohoo]

---

@stonent01 wrote:

Version 16.12.05b

---

Post the complete output to the following command: 

1. sh version
2. sh platform resources
3. sh platform software status con brief

[stonent01]

nswb3850#show ver
Cisco IOS XE Software, Version 16.12.05b
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.12.5b, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Thu 25-Mar-21 13:09 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2021 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 4.78, RELEASE SOFTWARE (P)

nswb3850 uptime is 1 year, 7 weeks, 7 hours, 52 minutes
Uptime for this control processor is 1 year, 7 weeks, 7 hours, 54 minutes
System returned to ROM by Power Failure or Unknown at 23:40:10 CST Sat Oct 2 2021
System restarted at 23:52:10 CST Sat Oct 2 2021
System image file is "flash:packages.conf"
Last reload reason: Power Failure or Unknown

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
lanbasek9 Smart License lanbasek9
None Subscription Smart License None

Smart Licensing Status: UNREGISTERED/EVAL EXPIRED

cisco WS-C3850-48T (MIPS) processor (revision AA0) with 794888K/6147K bytes of memory.
Processor board ID (Removed before posting)
2 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
1611414K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : (Removed before posting)
Motherboard Assembly Number : 73-16296-08
Motherboard Serial Number : (Removed before posting)
Model Revision Number : AA0
Motherboard Revision Number : A0
Model Number : WS-C3850-48T
System Serial Number : (Removed before posting)

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 WS-C3850-48T 16.12.05b CAT3K_CAA-UNIVERSALK9 INSTALL

Configuration register is 0x102

nswb3850#show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 10.47% 100% 90% 95% H
DRAM 1654MB(43%) 3842MB 90% 95% H
TMPFS 139MB(3%) 3842MB 40% 50% H

nswb3850#show platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.63 0.53 0.47

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3934800 1694072 (43%) 2240728 (57%) 2652768 (67%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 3.90 0.70 0.00 95.39 0.00 0.00 0.00
1 7.60 1.30 0.00 91.09 0.00 0.00 0.00
2 7.80 1.10 0.00 91.09 0.00 0.00 0.00
3 10.98 2.39 0.00 86.51 0.00 0.09 0.00

nswb3850#

[Georg Pauwen]

Hello,

post the running configuration (sh run) of one of the affected switches...

[Leo Laohoo]

Output looks good. 

How many clients are using this SVI? 

[stonent01]

I know it's been a few months, but I wanted to update everyone.  I got the problem resolved.  I was able to schedule some downtime in the datacenter and reboot that switch and the problem immediately went away after the reboot.

I'm glad it's over, I was getting tired of the annoying e-mails from our monitoring system that the switch was offline.

[balaji.bandi]

Sure i see switch also high uptime

nswb3850 uptime is 1 year, 7 weeks, 7 hours, 52 minutes

Thank you for sharing the information and it is now resolved, we mark this as resolved now.

Sure NMS give that alerts, before reboot, you would have unmanage to get that dirty emails filling the box.