is it safe to run debug commands on high cpu utilization switch?

[Herman2018]

Hi, can anyone advise whether it is saft to run debug commands "debug platform software memory <process> <R0/R1/RP xxx > alloc callsite start "  and "debug platform software memory .... alloc backtrace start "  ? on c9407r switch?

we saw the error log on the switch. Is it software bug? the XE OS version is 16.12.1r. 

PLATFORM-3-ELEMENT_TMPFS_CRITICAL: Chassis 1 R0/0: smand: 1/RP/0: TMPFS value 70% above critical level 50%
 %PLATFORM-3-ELEMENT_CRITICAL: Chassis 1 R0/0: smand: 1/RP/0: Used Memory value 97% exceeds critical level 95%

Thanks in advance. 

[marce1000 · Accepted solution]

 - FYI :  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd99665
            https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvy76922

  M.
             

[Filip Po]

Hello.

In general, debug commands should be run in a lab environment first. However, if you need to run the debug in a production environment, then definitely only during maintenance time and possibly with the device reload deferred in case and output captured to a file rather than to the console.

[Herman2018]

thanks @Filip Po a lot for your advice!

[Leo Laohoo]

---

@Herman2018 wrote:
PLATFORM-3-ELEMENT_TMPFS_CRITICAL: Chassis 1 R0/0: smand: 1/RP/0: TMPFS value 70% above critical level 50%

---

TMPFS?  What is the firmware is this controller running on?  

NOTE:  12.2(1r) is NOT an IOS-XE version but ROMMON version.  Two different things. 

Post the complete output to the following commands:

sh platform software mount switch active r0 | include ^tmpfs.*tmp | exclude /tmp/

[Herman2018]

@Leo Laohoo thanks for advice. yes, it is related to TMPFS, the OS version 16.12.1r. I will post the output of the command later. 

[Leo Laohoo]

---

@Herman2018 wrote:
the OS version 16.12.1r.

---

@Herman2018, 

Please read my responses carefully.  16.12.1r is the ROMMON version and not what I am asking for.  

What firmware is the chassis running on?  If unsure, post the complete output to the command "sh version".

[Herman2018]

Thanks @Leo Laohoo . sorry, the software version is 16.12.02. It is a chassis. 

[Leo Laohoo · Accepted solution]

The entire 16.12.X is known to be very buggy and riddled with memory leaks.  

Upgrade to 17.9.3 (and later) and see if this improve things.

[Herman2018]

Thanks @Leo Laohoo a lot for your kind and prompt advices!!! Appreciated !

[Joseph W. Doherty]

"is it safe to run debug commands on high cpu utilization switch?"

Well that depends much on what you consider "safe".

Debug commands often load up the CPU, and if you push your CPU to 100%, often you start to run into "issues".  As you mention CPU is already high, proceed very, very (very) carefully, with the understanding you might be adding "the (proverbial) straw that broke the camel's back".

BTW, part of proceeding very carefully, is trying to use any debug commands, such that whatever information you're after, is obtained with the least "intrusive" debug option(s).  For example, you might not want to start using debug with its "all" option.  ; )

@Filip Po suggests running debug first in the lab, and if done on a production system, during a maintenance period.  Both good suggestions, but, alas, often whatever we're trying to debug might not reveal itself then.

One very nice use, though, of trying debug in a lab first, is to explore the various debug options and their possible impact.  Possible such can be used to find a way to debug with minimal impact.

He also suggests to save output to a file rather than the console.  If he means output to the "real" console port, he may (still) be on point.  (Cisco used to [still?] document using the actual console port was resource intensive [being a serial RS-232 port].)

If you're using something like telnet, I don't believe (?) it would be much more, if at all, more impactful than writing output to a file.

[Herman2018]

Thanks @Joseph W. Doherty for your kind advice!