CSCvs94143 - IOS shim client ngwc fed bipc' has taken msec (runtime 0 msec) to process a 'unknown' ...

johnha · ‎06-15-2021

We have attempted "install add file flash://cat9k_lite_iosxe.16.12.04.SPA.bin" as part of switch upgrade. got the following message. model # C9200L-48P-4X (2xSwitches stacked) / current version: 16.9.5

Jun 16 10:28:35: %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'sman dc 3 bipc' has taken 2404 msec (runtime: 1880 msec) to process a 'do_interactive' message
Jun 16 10:36:37: %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'ngwc fed bipc' has taken 432 msec (runtime: 404 msec) to process a 'unknown' message

Leo Laohoo · ‎06-15-2021

Try the following command:

install add file bootflash:cat9k_lite_iosxe.16.12.04.SPA.bin activate commit

WARNING: This will automatically reboot the stack.

sphillips · ‎08-22-2023

What is the resolve for this bug (CSCvs94143) ? I keep losing my management vlan because this bug spikes the CPU and causes it to go down. It's happened a few times over the past week and today it cause a reboot of the stack which caused a 5 minute outage. Is there a workaround or is an upgrade the only resolution? I believe we're running 16.9.3. Does anyone know if the problem happens on version 17.x.x?

Leo Laohoo · ‎08-22-2023

@sphillips wrote:
I believe we're running 16.9.3.

Double check and do not make presumption. What firmware is the stack running on?

Post the complete output to the following commands:

sh platform resources
sh platform software status conn brief
sh crypto pki trustpool | i cn=

sphillips · ‎08-23-2023

Yes we're running 16.9.3. I had trouble with one of the commands but here is the output you've requested.

MDF_Stack#sh platform rescou ources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 3.28% 100% 90% 95% H
DRAM 1082MB(55%) 1952MB 90% 95% H

MDF_Stack#sh platform c softwat re w status control -processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Unknown 0.23 0.24 0.18
2-RP0 Unknown 0.09 0.04 0.03
3-RP0 Unknown 0.12 0.13 0.09

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 1998932 1108388 (55%) 890544 (45%) 1514584 (76%)
2-RP0 Healthy 1998932 1084312 (54%) 914620 (46%) 1480572 (74%)
3-RP0 Healthy 1998932 804236 (40%) 1194696 (60%) 815372 (41%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 2.30 1.04 0.00 96.54 0.00 0.10 0.00
1 2.52 1.58 0.00 95.78 0.00 0.10 0.00
2 3.24 1.04 0.00 95.70 0.00 0.00 0.00
3 2.81 1.04 0.00 96.13 0.00 0.00 0.00
2-RP0 0 1.76 0.51 0.00 97.60 0.00 0.10 0.00
1 1.66 0.93 0.00 97.40 0.00 0.00 0.00
2 1.66 0.51 0.00 97.81 0.00 0.00 0.00
3 1.44 0.61 0.00 97.93 0.00 0.00 0.00
3-RP0 0 1.74 0.72 0.00 97.53 0.00 0.00 0.00
--More-- 1 1.44 0.82 0.00 97.72 0.00 0.00 0.00
2 1.34 0.62 0.00 98.02 0.00 0.00 0.00
3 1.73 1.02 0.00 97.24 0.00 0.00 0.00

MDF_Stack#sh crypto pki trusd stpool | i cn=
cn=QuoVadis Root CA 2
cn=QuoVadis Root CA 2
cn=VeriSign Class 3 Public Primary Certification Authority - G5
cn=VeriSign Class 3 Public Primary Certification Authority - G5
cn=Cisco RXC-R2
cn=Cisco RXC-R2
cn=Cisco Root CA 2048
cn=ACT2 SUDI CA
cn=Licensing Root - DEV
cn=Licensing Root - DEV
cn=Cisco Root CA M2
cn=Cisco Manufacturing CA SHA2
cn=Cisco Root CA M2
cn=Cisco Root CA M2
cn=Cisco Root CA 2048
cn=Cisco Manufacturing CA
cn=Cisco Root CA 2048
cn=Cisco Root CA 2048
cn=Cisco Root CA M1
cn=Cisco Root CA M1
cn=Cisco Licensing Root CA
cn=Cisco Licensing Root CA
MDF_Stack#ex

Leo Laohoo · ‎08-23-2023

@sphillips wrote:
MDF_Stack#sh crypto pki trustpool | i cn=
cn=QuoVadis Root CA 2
cn=QuoVadis Root CA 2

FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality

sphillips · ‎08-24-2023

Thank you for the information. The MDF stack showed reason for last reload is critical process smand fault on rp_0_0 (rc=139). We had been getting the same symptoms on edge switch and they were rebooting as well. Upgraded firmware on the edge switches to 17.6.4. We skipped this MDF stack due to not wanting downtime but appears we need to upgrade firmware to 17.6.4. Would you agree?

Leo Laohoo · ‎08-24-2023

I made a mistake of forgetting to add this: FN - 72578 - Cisco IOS XE - Smart Licensing Using Policy Might Cause High CPU/Memory Usage

The words in this field notice is designed to fool or downplay the effects. If those CPU hogs &/or memory leak are left unattended, there is only on consequence and that is a crash.

sphillips · ‎08-24-2023

Good info. I honestly think I'm seeing 2 different bugs. The first bug reboots my stack and the second bug drops my 10g uplink management VLAN.

The first bug caused the stack to reboot is CAT9K reload due to "Critical process cmand fault on rp_0_0 (rc=139)" CSCvv86246

Symptom: Member switch of a Stackwise virtual cluster may reload with reload reason of: Critical process cmand fault on rp_0_0 (rc=139)

Update firmware version to 17.6.4 should resolve it.

The second bug drops my 10g Uplink for my management Vlan. The bug description is CSCvs94143

IOS shim client 'ngwc fed bipc' has taken msec (runtime: 0 msec) to process a 'unknown' message.

Symptom: High cpu spike is observed with on switch with the following log being generated, %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'ngwc fed bipc' has taken ### msec (runtime: 0 msec) to process a 'unknown' "###" would display the number of msec that the process was held. This CPU spike would then lead to issues to processing control-plane traffic.

This affected firmware version 16.9.3 which is what I'm running but the bug tool doesn't show a known fixed release.

CSCvs94143 - IOS shim client ngwc fed bipc' has taken msec (runtime 0 msec) to process a 'unknown' message - 4