High CPU Utilization - 6500 Switch - PID 872 NF AM App Push M

[herohayah]

Hello,

     I'm using a 6500 Switch as the CORE router in a 3 tiered Network.  Lately, the 6500 has been spiking with High CPU Utilization and it tends to bring down the network.  Noticed that whenever the CPU spikes, PID 872 NF AM App Push M is always at the top of the CPU process list.  I have tried to research what this is but there is no mention of it on Google.  Was wondering if anyone knows what this is and how I might be able to mitigate/stop it?

[Reza Sharifi]

Hi,

Can you post the output of "sh proce cpu sort | ex 0.00"?

HTH

[herohayah]

 Im no longer at work but I took a picture of "show proc CPU sort" while I was there.

[Reza Sharifi]

7.79 for 5 min does not seem to be very high for NF AM App. Do you have logs of the switch before and after the crash? 

Also, is there a crash file on the flash?

[herohayah]

It wasn't crashing when I took that photo.

I can get you more information when I return to work tomorrow.

Do you know what process NF AM App Push M is referring to?

[Leo Laohoo]

Post the complete output to the following commands: 

show version
show proc cpu history

[herohayah]

CAF-ITN1-CORE#show ver
Cisco IOS Software, s2t54 Software (s2t54-ADVIPSERVICESK9-M), Version 15.5(1)SY5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 04-Feb-20 14:05 by prod_rel_team

ROM: System Bootstrap, Version 12.2(50r)SYS2, RELEASE SOFTWARE (fc1)

CAF-ITN1-CORE uptime is 1 week, 1 day, 16 hours, 2 minutes
Uptime for this control processor is 1 week, 1 day, 16 hours, 2 minutes
System returned to ROM by Stateful Switchover
System image file is "bootdisk:s2t54-advipservicesk9-mz.SPA.155-1.SY5.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6506-E (M8572) processor (revision ) with 3162112K/524288K bytes of memory.
Processor board ID SAL1551Z0HF
CPU: MPC8572_E, Version: 2.2, (0x80E80022)
CORE: E500, Version: 3.0, (0x80210030)
CPU:1500MHz, CCB:600MHz, DDR:600MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

Last reset from s/w reset
19 Virtual Ethernet interfaces
102 Gigabit Ethernet interfaces
20 Ten Gigabit Ethernet interfaces
2543K bytes of non-volatile configuration memory.

Configuration register is 0x2102

CAF-ITN1-CORE#sho proc cpu sort | ex 0.00
CPU utilization for five seconds: 31%/8%; one minute: 23%; five minutes: 23%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
872 62770868 62088923 1010 10.55% 5.45% 5.06% 0 NF AM App Push M
9 6107388 367341 16625 5.59% 0.97% 0.77% 0 Check heaps
202 12171868 66600430 182 1.75% 1.55% 1.47% 0 slcp process
845 17992276 13591096 1323 0.95% 1.26% 1.16% 0 IP NAT Ager
25 7622380 82726578 92 0.71% 0.78% 0.68% 0 ARP Input
456 21735924 223889786 97 0.71% 1.35% 1.62% 0 IP Input
763 2049408 22718887 90 0.55% 0.32% 0.26% 0 Port manager per
533 3668932 580938 6315 0.55% 0.29% 0.28% 0 Hardware API bac
846 744524 4723742 157 0.31% 0.39% 0.41% 0 DHCPD Receive
722 3445920 813923 4233 0.15% 0.46% 0.41% 0 Env Poll
432 594700 1154690 515 0.15% 0.08% 0.06% 0 QOS Stats Gather
513 3378000 1178872 2865 0.15% 0.28% 0.27% 0 ADJ background
200 476660 7454506 63 0.15% 0.05% 0.05% 0 Online Linecard
506 681168 394686 1725 0.15% 0.03% 0.04% 0 XDR receive
898 926524 4311216 214 0.07% 0.02% 0.05% 0 SNMP ENGINE
398 350520 829373 422 0.07% 0.06% 0.05% 0 Lif stats RP tas
907 1628564 22239971 73 0.07% 0.17% 0.18% 0 DiagCard4/-1
235 579804 10599474 54 0.07% 0.09% 0.08% 0 ADJ resolve proc
817 361028 234803 1537 0.07% 0.04% 0.04% 0 LTL MGR cc

[Leo Laohoo]

That is not the output I requested.

[herohayah]

My apologies.  I was very busy at the time and misread your message

[herohayah]

[Leo Laohoo]

There is nothing wrong with the CPU.  

[herohayah]

Crashed about 10mins ago.  I grabbed this snip during the crash.  As you can see that "NF AM App Push M" process is high during the crash.  This always happens.  Any idea what "NF AM App Push M" is referring to?

[MHM Cisco World]

For the first 

Remove netflow

And share config of NAT

MHM

[herohayah]

***UPDATE***

We decided to just replace the 6500 with a 9600.  We wanted to have the 9600 just be the CORE (Housing the SVIs, NAT Translations, and provide upstream connectivity for the DISTROs).   

We ran into a separate issue though.  We were not able to ping google.com and have interVLAN routing work at the same time.  For some reason, once we input "ip routing" we could ping internally but couldn't ping google.  If we input "no ip routing" we could ping google but couldn't ping internally.  

Does this sound familiar to anyone or is this a known issue with the 9600?  Is there a known fix?

[MHM Cisco World]

Share 

Show ip route 

MHM