StackWise virtual standby RP high memory utilization

Stepan-Karel · ‎10-31-2024

Hi community,

I have stumbled upon an interesting problem. The Standby RP, switch 1 currently, is suffering from high memory utilization which could only be discovered by Cisco DNAC and this command:

show platform software status control-processor.

The output tells me the memory utilization of slot1-RP0 is 83%, while slot2-RP0 stays as low as 26%.

It is quite unfortunate as any other show memory - like commands output utilization of around 23% - my guess is that it is because they are executed by the active RP, which's utilization is around that number.

The hardware used is Catalyst 9500 running IOS-XE 17.06.04

Does anyone have any experience or advice on how to deal with this?

Flavio Miranda · ‎10-31-2024

@Stepan-Karel

Digging in some cisco docs, I have found this. Does your router is under heavy traffic or with lots of ACL?

Overall Control Plane Resources

Control plane memory and CPU utilization on each control processor allows you to keep a tab on the overall control plane resources. You can use the show platform software status control-processor brief command (summary view) or the show platform software status control-processor command (detailed view) to view control plane memory and CPU utilization information.

All control processors should show status, Healthy. Other possible status values are Warning and Critical. Warning indicates that the router is operational, but that the operating level should be reviewed. Critical implies that the router is nearing failure.

If you see a Warning or Critical status, take the following actions:

Reduce the static and dynamic loads on the system by reducing the number of elements in the configuration or by limiting the capacity for dynamic services.
Reduce the number of routes and adjacencies, limit the number of ACLs and other rules, reduce the number of VLANs, and so on.

Stepan-Karel · ‎11-01-2024

Hi Flavio,

Thank you for your answer, however I do not think that this problem is caused naturally by heavy load in the system, since the active RP, which produces most of it, keeps its memory utilization low.

There is an access-list that is logging, which produces a lot of console output, but again - I'm not sure this would utilize the standby chassis had this been the case...

The commands you have provided are the ones I had used before to confirm that the memory utilization on standby RP is high.

Thank you,

Stepan.

Leo Laohoo · ‎10-31-2024

@Stepan-Karel wrote:
slot1-RP0 is 83%

Slot 1 is not the standby.

There are several options/workaround:

1. Failover

2. Reboot Slot1

3. Reboot both units

4. Upgrade the firmware.

The "normal" memory utilization for a 9500 (standalone or VSS) is <35%. And the IOS-XE OS memory leaks like a sieve!

I strongly recommend to any router, switch, WLC with IOS-XE as an OS to regularly reboot their router(s), switches and WLC. Reboot the routers, switches &/or WLC every 6 or 12 months.

Stepan-Karel · ‎11-01-2024

Hi Leo,

How can you tell so surely that slot 1 is not the standby RP?

Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
1 Standby cced.4db9.xxxx 10 V02 Ready
*2 Active 806a.0009.xxxx 5 V02 Ready

Although the first chassis has higher switch priority, for historical failover reasons the active chassis is currently the second one.

Also we would like to avoid rebooting the system, hence why I'm here.

Thank you,

Stepan.

Leo Laohoo · ‎11-01-2024

@Stepan-Karel wrote:
How can you tell so surely that slot 1 is not the standby RP?

Mac persistency wait time: Indefinite
H/W Current
Switch# Role       Mac Address     Priority Version State
-------------------------------------------------------------------------------------
1      Standby cced.4db9.xxxx   10         V02    Ready

Because it says so right there. Slot 1, Standby.